2024-08-06T17:30:34.721 INFO:root:teuthology version: 0.0.1.dev388+g4bc527c 2024-08-06T17:30:34.740 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-06T17:30:34.844 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173 branch: main description: rados:thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} email: null first_in_suite: false job_id: '7840173' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'rbd object class functional tests ' name: lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 30 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: global: mon client directed command retry: 5 mon election default strategy: 1 ms inject socket failures: 2500 ms tcp read timeout: 5 osd_pool_default_min_size: 2 osd_pool_default_size: 2 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 paxos service trim min: 10 osd: bluestore zero block detection: true debug ms: 1 debug osd: 20 osd blocked scrub grace period: 3600 osd debug verify cached snaps: true osd debug verify missing on start: true osd map cache size: 1 osd max backfills: 6 osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd op queue: debug_random osd op queue cut off: debug_random osd scrub during recovery: false osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(REQUEST_STUCK\) - \(MON_DOWN\) - \(OSD_SLOW_PING_TIME - but it is still running - objects unfound and apparently lost - osd_map_cache_size - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 workunit: branch: main sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 owner: scheduled_lflores@folio02 priority: 60 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - - mon.c - osd.8 - osd.9 - osd.10 - osd.11 - client.2 seed: 3565 sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 sleep_before_teardown: 0 subset: 111/120000 suite: rados:thrash-old-clients suite_branch: main suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 0325fd3c11a225c0fd9eb38fab88aad1370ad779 targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBT0YOP9dW4CVLug48UEIvLppzfyOCfQ7/hz3EWPZ7pYGS1lqsZQq62ZnNPooV2x7pj44tcDI7V9QM6Z9DW2a30= smithi107.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAL8sNfn8ff10fBy/gj2XPS04ohTZ1jvJjW2F91M2tpmZR3raYnHp+At7/tMaSdTYVH+AH6zDq7WSiSzMBrqet4= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDvT9raAXRNKejxdTIwzmUjrIM3Ks9qyBru6xJGszqsMGhUYoPJ0Il1owZk/8kAKEp/vZcb/5edYKAdu1alHdg= tasks: - install: branch: quincy exclude_packages: - ceph-mgr-dashboard - ceph-mgr-diskprediction-local - ceph-mgr-rook - ceph-mgr-cephadm - ceph-base-debuginfo - ceph-common-debuginfo - ceph-immutable-object-cache-debuginfo - ceph-radosgw-debuginfo - ceph-test-debuginfo - ceph-base-debuginfo - ceph-mgr-debuginfo - ceph-mds-debuginfo - ceph-mon-debuginfo - ceph-osd-debuginfo - ceph-fuse-debuginfo - librados-devel-debuginfo - libcephfs2-debuginfo - librados2-debuginfo - librbd1-debuginfo - python3-cephfs-debuginfo - python3-rados-debuginfo - python3-rbd-debuginfo - python3-rgw-debuginfo - rbd-fuse-debuginfo - rbd-mirror-debuginfo - rbd-nbd-debuginfo - ceph-volume - cephadm: conf: mon: auth allow insecure global id reclaim: true - exec: mon.a: - while ! ceph balancer status ; do sleep 1 ; done - ceph balancer mode crush-compat - ceph balancer on - thrashosds: chance_pgnum_grow: 0.25 chance_pgpnum_fix: 0.25 chance_test_map_discontinuity: 2 timeout: 1800 - exec: client.2: - ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 4bc527c722abc094dbc33e04519a3647224076f6 timestamp: 2024-08-06_17:19:44 tube: smithi user: lflores verbose: true verify_ceph_hash: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.4074525 2024-08-06T17:30:34.844 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa; will attempt to use it 2024-08-06T17:30:34.845 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks 2024-08-06T17:30:34.845 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-08-06T17:30:34.847 INFO:teuthology.task.internal:Saving configuration 2024-08-06T17:30:34.861 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-08-06T17:30:34.919 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-08-06T17:30:34.951 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-06 17:25:18.589066', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBT0YOP9dW4CVLug48UEIvLppzfyOCfQ7/hz3EWPZ7pYGS1lqsZQq62ZnNPooV2x7pj44tcDI7V9QM6Z9DW2a30='} 2024-08-06T17:30:34.984 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi107.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-06 17:25:18.590487', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAL8sNfn8ff10fBy/gj2XPS04ohTZ1jvJjW2F91M2tpmZR3raYnHp+At7/tMaSdTYVH+AH6zDq7WSiSzMBrqet4='} 2024-08-06T17:30:35.015 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-08-06 17:25:18.587420', 'locked_by': 'scheduled_lflores@folio02', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEDvT9raAXRNKejxdTIwzmUjrIM3Ks9qyBru6xJGszqsMGhUYoPJ0Il1owZk/8kAKEp/vZcb/5edYKAdu1alHdg='} 2024-08-06T17:30:35.015 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-08-06T17:30:35.022 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-08-06T17:30:35.022 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1'] 2024-08-06T17:30:35.022 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.c', 'osd.8', 'osd.9', 'osd.10', 'osd.11', 'client.2'] 2024-08-06T17:30:35.022 INFO:teuthology.run_tasks:Running task console_log... 2024-08-06T17:30:35.135 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fdc88f6fac0>, signals=[15]) 2024-08-06T17:30:35.135 INFO:teuthology.run_tasks:Running task internal.connect... 2024-08-06T17:30:35.141 INFO:teuthology.task.internal:Opening connections... 2024-08-06T17:30:35.141 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-08-06T17:30:35.143 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:30:35.235 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2024-08-06T17:30:35.236 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:30:35.325 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-08-06T17:30:35.326 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:30:35.410 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-08-06T17:30:35.417 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-08-06T17:30:35.435 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-08-06T17:30:35.435 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-08-06T17:30:35.493 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-06T17:30:35.494 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-08-06T17:30:35.494 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-06T17:30:35.494 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-06T17:30:35.494 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-06T17:30:35.494 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-08-06T17:30:35.527 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2024-08-06T17:30:35.545 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2024-08-06T17:30:35.545 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Stream" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="9" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="9" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el9" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-06T17:30:35.602 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:LOGO="fedora-logo-icon" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://centos.org/" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-06T17:30:35.603 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-06T17:30:35.603 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2024-08-06T17:30:35.636 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-08-06T17:30:35.653 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-08-06T17:30:35.654 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-08-06T17:30:35.711 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-08-06T17:30:35.712 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-08-06T17:30:35.712 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-08-06T17:30:35.712 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-08-06T17:30:35.712 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-08-06T17:30:35.712 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-08-06T17:30:35.712 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-08-06T17:30:35.744 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-08-06T17:30:35.755 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-08-06T17:30:35.783 INFO:teuthology.task.internal:Checking for old test directory... 2024-08-06T17:30:35.783 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-08-06T17:30:35.786 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2024-08-06T17:30:35.788 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-08-06T17:30:35.804 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-08-06T17:30:35.810 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-08-06T17:30:35.810 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-08-06T17:30:35.842 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2024-08-06T17:30:35.845 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-08-06T17:30:35.940 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-08-06T17:30:36.042 INFO:teuthology.run_tasks:Running task kernel... 2024-08-06T17:30:36.057 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-08-06T17:30:36.058 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}, 'mon.c': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-08-06T17:30:36.058 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-06T17:30:36.059 DEBUG:teuthology.orchestra.run.smithi107:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-06T17:30:36.059 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-08-06T17:30:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:30:36.076 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-08-06T17:30:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:30:36.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:30:36.077 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-08-06T17:30:36.078 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-08-06T17:30:36.133 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-437.el9.x86_64 2024-08-06T17:30:36.133 INFO:teuthology.task.kernel:Running kernel on smithi107: 5.14.0-437.el9.x86_64 2024-08-06T17:30:36.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2024-08-06T17:30:36.135 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-437.el9.x86_64 2024-08-06T17:30:36.135 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-437.el9.x86_64 2024-08-06T17:30:36.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-08-06T17:30:36.137 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-437.el9.x86_64 2024-08-06T17:30:36.137 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.14.0-437.el9.x86_64 2024-08-06T17:30:36.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum install -y kernel 2024-08-06T17:30:39.025 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 6.4 MB/s | 8.2 MB 00:01 2024-08-06T17:30:39.025 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.2 MB 00:01 2024-08-06T17:30:39.284 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 5.6 MB/s | 8.2 MB 00:01 2024-08-06T17:30:41.467 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-08-06T17:30:43.017 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 6.9 MB/s | 20 MB 00:02 2024-08-06T17:30:44.893 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 4.4 MB/s | 20 MB 00:04 2024-08-06T17:30:47.109 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.6 MB 00:01 2024-08-06T17:30:48.850 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.6 MB 00:01 2024-08-06T17:30:49.185 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.6 MB 00:00 2024-08-06T17:30:49.394 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 18 kB 00:01 2024-08-06T17:30:50.355 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-06T17:30:50.954 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 18 kB 00:00 2024-08-06T17:30:51.010 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 20 kB/s | 18 kB 00:00 2024-08-06T17:30:51.965 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 25 MB/s | 22 MB 00:00 2024-08-06T17:30:51.988 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 26 MB/s | 22 MB 00:00 2024-08-06T17:30:55.719 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-08-06T17:30:57.279 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 42 kB/s | 1.7 kB 00:00 2024-08-06T17:30:57.343 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-08-06T17:30:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-06T17:30:57.733 INFO:teuthology.orchestra.run.smithi117.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-06T17:30:57.824 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-06T17:30:57.832 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-08-06T17:30:57.833 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-06T17:30:57.833 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-06T17:30:57.833 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-06T17:30:57.833 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:30:57.833 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 84 M 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 126 M 2024-08-06T17:30:57.834 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:30:58.136 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 6.2 MB/s | 1.5 MB 00:00 2024-08-06T17:30:58.437 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 32 MB/s | 17 MB 00:00 2024-08-06T17:30:58.812 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 44 MB/s | 30 MB 00:00 2024-08-06T17:30:59.129 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 29 MB/s | 36 MB 00:01 2024-08-06T17:30:59.129 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:30:59.129 INFO:teuthology.orchestra.run.smithi117.stdout:Total 65 MB/s | 84 MB 00:01 2024-08-06T17:30:59.352 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-06T17:30:59.352 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-06T17:30:59.357 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-08-06T17:30:59.357 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-08-06T17:30:59.445 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:30:59.449 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:30:59.451 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:30:59.451 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:30:59.452 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:30:59.453 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-08-06T17:30:59.453 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:30:59.453 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 84 M 2024-08-06T17:30:59.453 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 126 M 2024-08-06T17:30:59.453 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 5.14.0-480.el9 baseos 1.5 M 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 5.14.0-480.el9 baseos 17 M 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 5.14.0-480.el9 baseos 36 M 2024-08-06T17:30:59.456 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core x86_64 5.14.0-480.el9 baseos 30 M 2024-08-06T17:30:59.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:30:59.457 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:30:59.457 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:30:59.457 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-08-06T17:30:59.457 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:30:59.458 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 84 M 2024-08-06T17:30:59.458 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 126 M 2024-08-06T17:30:59.458 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:30:59.805 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 5.2 MB/s | 1.5 MB 00:00 2024-08-06T17:30:59.859 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:31:00.208 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:31:00.208 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:31:00.422 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-480.el9.x86_64.rpm 19 MB/s | 17 MB 00:00 2024-08-06T17:31:00.819 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-5.14.0-480.el9.x86_64.rpm 1.2 MB/s | 1.5 MB 00:01 2024-08-06T17:31:00.948 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-core-5.14.0-480.el9.x86_6 26 MB/s | 30 MB 00:01 2024-08-06T17:31:01.189 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:31:01.190 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:31:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 21 MB/s | 36 MB 00:01 2024-08-06T17:31:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:31:01.257 INFO:teuthology.orchestra.run.smithi086.stdout:Total 47 MB/s | 84 MB 00:01 2024-08-06T17:31:01.989 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:31:02.336 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:31:02.336 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:31:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:31:03.352 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:31:03.632 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:31:04.336 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:31:04.429 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:04.519 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-modules-core-5.14.0-480.el9.x86_6 8.1 MB/s | 30 MB 00:03 2024-08-06T17:31:05.777 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:31:06.277 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-core-5.14.0-480.el9.x86_64.rpm 2.6 MB/s | 17 MB 00:06 2024-08-06T17:31:06.513 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:31:06.606 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:07.075 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:07.214 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:09.281 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:09.425 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:12.047 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:12.131 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:14.242 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:14.392 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:17.004 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:19.241 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:37.397 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): kernel-modules-5.14.0-480.el9.x86_64.rpm 973 kB/s | 36 MB 00:37 2024-08-06T17:31:37.397 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:31:37.397 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.2 MB/s | 84 MB 00:37 2024-08-06T17:31:38.124 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:31:38.451 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:31:38.451 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:31:39.467 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:31:39.467 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:31:41.916 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:31:42.604 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-core-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:31:42.697 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:45.361 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:45.509 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:46.942 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:46.983 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:47.285 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:47.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:31:47.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:47.285 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:31:48.059 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:31:48.401 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:31:48.484 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.484 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:31:48.485 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-06T17:31:48.485 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-06T17:31:48.485 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-06T17:31:48.485 INFO:teuthology.task.kernel:Installing distro kernel on mon.c... 2024-08-06T17:31:48.485 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi117.front.sepia.ceph.com, path=None, version=distro) 2024-08-06T17:31:48.485 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:31:48.543 INFO:teuthology.orchestra.run.smithi117.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:48.543 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:31:48.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo rpm -qi grub2-tools 2024-08-06T17:31:48.581 INFO:teuthology.orchestra.run.smithi117.stdout:Name : grub2-tools 2024-08-06T17:31:48.581 INFO:teuthology.orchestra.run.smithi117.stdout:Epoch : 1 2024-08-06T17:31:48.581 INFO:teuthology.orchestra.run.smithi117.stdout:Version : 2.06 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Release : 68.el9 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: x86_64 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Group : Unspecified 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Size : 8264254 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:License : GPLv3+ 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Packager : builder@centos.org 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:Vendor : CentOS 2024-08-06T17:31:48.582 INFO:teuthology.orchestra.run.smithi117.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:Summary : Support tools for GRUB. 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:Description : 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:hardware devices. 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:31:48.583 INFO:teuthology.orchestra.run.smithi117.stdout:This subpackage provides tools for support of all platforms. 2024-08-06T17:31:48.584 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-06T17:31:48.584 INFO:teuthology.task.kernel:Updating grub on smithi117 to boot 5.14.0-480.el9.x86_64 2024-08-06T17:31:48.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-06T17:31:48.815 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:48.855 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:31:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:31:49.175 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:49.199 INFO:teuthology.orchestra.run.smithi117.stderr:Generating grub configuration file ... 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-06T17:31:49.969 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-06T17:31:49.970 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-06T17:31:49.970 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:31:49.970 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:31:50.354 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:31:50.371 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:31:50.433 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:31:50.454 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:50.454 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:31:50.454 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-06T17:31:50.454 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-06T17:31:50.454 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-06T17:31:50.455 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-08-06T17:31:50.455 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-08-06T17:31:50.455 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:31:50.512 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:31:50.512 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:31:50.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-08-06T17:31:50.542 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-08-06T17:31:50.542 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-08-06T17:31:50.543 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:31:50.544 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-08-06T17:31:50.545 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-06T17:31:50.545 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-480.el9.x86_64 2024-08-06T17:31:50.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-06T17:31:50.590 INFO:teuthology.orchestra.run.smithi117.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-06T17:31:50.620 INFO:teuthology.orchestra.run.smithi117.stderr:done 2024-08-06T17:31:50.623 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2024-08-06T17:31:50.637 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.yANL5SJ94z 2024-08-06T17:31:50.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.yANL5SJ94z 2024-08-06T17:31:50.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.yANL5SJ94z 2024-08-06T17:31:50.839 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.yANL5SJ94z is 6KB 2024-08-06T17:31:50.859 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.yANL5SJ94z 2024-08-06T17:31:50.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo /bin/ls /boot/loader/entries || true 2024-08-06T17:31:50.934 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-06T17:31:50.934 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-06T17:31:50.935 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-06T17:31:50.935 INFO:teuthology.orchestra.run.smithi117.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-06T17:31:50.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-06T17:31:51.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo shutdown -r now 2024-08-06T17:31:51.114 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-08-06T17:31:52.540 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-06T17:31:52.577 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-08-06T17:31:52.579 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-08-06T17:31:52.594 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.edJfOdABpj 2024-08-06T17:31:52.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.edJfOdABpj 2024-08-06T17:31:52.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.edJfOdABpj 2024-08-06T17:31:52.811 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.edJfOdABpj is 6KB 2024-08-06T17:31:52.860 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.edJfOdABpj 2024-08-06T17:31:52.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-08-06T17:31:52.936 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-06T17:31:52.936 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-06T17:31:52.936 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-06T17:31:52.936 INFO:teuthology.orchestra.run.smithi086.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-06T17:31:52.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-06T17:31:53.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-08-06T17:31:55.308 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:32:21.086 INFO:teuthology.task.kernel:Checking client mon.c for new kernel version... 2024-08-06T17:32:21.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:32:21.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:32:23.084 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-08-06T17:32:23.084 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:32:23.085 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:32:24.796 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:32:24.835 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:32:25.121 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:32:25.121 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-5.14.0-480.el9.x86_64 1/4 2024-08-06T17:32:25.121 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-5.14.0-480.el9.x86_64 2/4 2024-08-06T17:32:25.121 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-5.14.0-480.el9.x86_64 3/4 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-core-5.14.0-480.el9.x86_64 4/4 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-5.14.0-480.el9.x86_64 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-5.14.0-480.el9.x86_64 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-5.14.0-480.el9.x86_64 2024-08-06T17:32:25.973 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-core-5.14.0-480.el9.x86_64 2024-08-06T17:32:25.974 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:32:25.974 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:32:26.374 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:32:26.459 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:32:26.459 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:32:26.459 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-480.el9.x86_64 2024-08-06T17:32:26.459 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-08-06T17:32:26.459 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-08-06T17:32:26.459 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-08-06T17:32:26.460 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2024-08-06T17:32:26.460 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel | sort -rV | head -n 1 2024-08-06T17:32:26.516 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-5.14.0-480.el9.x86_64 2024-08-06T17:32:26.517 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-480.el9.x86_64 2024-08-06T17:32:26.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2024-08-06T17:32:26.547 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.06 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 68.el9 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Group : Unspecified 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 8264254 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-08-06T17:32:26.548 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : builder@centos.org 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2024-08-06T17:32:26.549 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:32:26.550 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2024-08-06T17:32:26.551 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-08-06T17:32:26.551 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 5.14.0-480.el9.x86_64 2024-08-06T17:32:26.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-08-06T17:32:27.156 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2024-08-06T17:32:28.435 INFO:teuthology.orchestra.run.smithi107.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-08-06T17:32:28.461 INFO:teuthology.orchestra.run.smithi107.stderr:done 2024-08-06T17:32:28.464 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2024-08-06T17:32:28.482 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.p2Kz5zyvFz 2024-08-06T17:32:28.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.p2Kz5zyvFz 2024-08-06T17:32:28.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.p2Kz5zyvFz 2024-08-06T17:32:28.693 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.p2Kz5zyvFz is 6KB 2024-08-06T17:32:28.741 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.p2Kz5zyvFz 2024-08-06T17:32:28.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2024-08-06T17:32:28.827 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-08-06T17:32:28.827 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-08-06T17:32:28.827 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-08-06T17:32:28.827 INFO:teuthology.orchestra.run.smithi107.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64.conf 2024-08-06T17:32:28.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-480.el9.x86_64 2024-08-06T17:32:28.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2024-08-06T17:32:39.518 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-08-06T17:32:41.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T17:32:48.526 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:32:48.527 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:32:50.574 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:32:50.574 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:32:51.615 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-08-06T17:32:53.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T17:32:58.978 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-08-06T17:32:58.979 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:32:58.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:03.618 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:33:03.619 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:05.669 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:33:05.670 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:17.407 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-08-06T17:33:22.018 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.117 2024-08-06T17:33:26.411 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:33:26.412 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:29.474 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-08-06T17:33:37.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:33:37.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:37.405 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-08-06T17:33:37.879 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-08-06T17:33:37.935 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:33:37.935 INFO:teuthology.task.kernel:Checking kernel version of mon.c, want "5.14.0-480.el9.x86_64"... 2024-08-06T17:33:37.936 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2024-08-06T17:33:37.991 INFO:teuthology.orchestra.run.smithi117.stdout:5.14.0-480.el9.x86_64 2024-08-06T17:33:37.991 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-06T17:33:37.992 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-06T17:33:37.992 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-06T17:33:38.326 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:33:38.326 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-480.el9.x86_64"... 2024-08-06T17:33:38.327 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-08-06T17:33:38.341 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-480.el9.x86_64 2024-08-06T17:33:38.342 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-06T17:33:38.342 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-06T17:33:38.342 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-06T17:33:38.993 INFO:teuthology.task.kernel:Enabling kdb on mon.c... 2024-08-06T17:33:38.993 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-06T17:33:39.206 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2024-08-06T17:33:39.252 DEBUG:teuthology.parallel:result is None 2024-08-06T17:33:39.343 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-08-06T17:33:39.343 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-06T17:33:39.424 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-08-06T17:33:39.463 DEBUG:teuthology.parallel:result is None 2024-08-06T17:33:41.475 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:33:41.476 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:33:59.907 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2024-08-06T17:34:14.907 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:34:14.909 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:34:15.333 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-08-06T17:34:15.717 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:34:15.717 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-480.el9.x86_64"... 2024-08-06T17:34:15.717 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2024-08-06T17:34:15.731 INFO:teuthology.orchestra.run.smithi107.stdout:5.14.0-480.el9.x86_64 2024-08-06T17:34:15.732 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-480.el9.x86_64 vs 5.14.0-480.el9.x86_64 2024-08-06T17:34:15.732 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-08-06T17:34:15.732 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-08-06T17:34:16.733 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-08-06T17:34:16.733 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-08-06T17:34:16.809 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2024-08-06T17:34:16.858 DEBUG:teuthology.parallel:result is None 2024-08-06T17:34:16.859 INFO:teuthology.run_tasks:Running task internal.base... 2024-08-06T17:34:16.866 INFO:teuthology.task.internal:Creating test directory... 2024-08-06T17:34:16.866 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-06T17:34:16.869 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-06T17:34:16.871 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-08-06T17:34:16.890 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-08-06T17:34:16.897 INFO:teuthology.run_tasks:Running task internal.archive... 2024-08-06T17:34:16.910 INFO:teuthology.task.internal:Creating archive directory... 2024-08-06T17:34:16.911 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-06T17:34:16.929 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-06T17:34:16.931 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-08-06T17:34:17.052 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-08-06T17:34:17.059 INFO:teuthology.task.internal:Enabling coredump saving... 2024-08-06T17:34:17.059 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-06T17:34:17.062 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-06T17:34:17.064 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-08-06T17:34:17.099 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.101 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.105 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.111 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.112 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.115 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-08-06T17:34:17.117 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-08-06T17:34:17.124 INFO:teuthology.task.internal:Configuring sudo... 2024-08-06T17:34:17.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-06T17:34:17.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-06T17:34:17.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-08-06T17:34:17.187 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-08-06T17:34:17.197 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-08-06T17:34:17.197 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-06T17:34:17.225 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-06T17:34:17.227 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-08-06T17:34:17.245 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.372 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:17.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:17.503 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:34:17.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-06T17:34:17.568 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.661 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:17.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:17.784 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:34:17.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-06T17:34:17.850 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-08-06T17:34:17.945 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:18.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-08-06T17:34:18.073 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:34:18.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-08-06T17:34:18.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-08-06T17:34:18.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2024-08-06T17:34:18.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-08-06T17:34:18.191 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T17:34:18.198 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T17:34:18.237 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T17:34:18.708 INFO:teuthology.run_tasks:Running task internal.timer... 2024-08-06T17:34:18.715 INFO:teuthology.task.internal:Starting timer... 2024-08-06T17:34:18.715 INFO:teuthology.run_tasks:Running task pcp... 2024-08-06T17:34:18.728 INFO:teuthology.run_tasks:Running task selinux... 2024-08-06T17:34:18.738 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-08-06T17:34:18.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-08-06T17:34:18.789 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-08-06T17:34:18.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2024-08-06T17:34:18.865 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: 2024-08-06T17:34:18.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-08-06T17:34:18.920 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-08-06T17:34:18.923 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-08-06T17:34:18.923 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-08-06T17:34:18.961 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-08-06T17:34:18.961 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2024-08-06T17:34:18.994 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2024-08-06T17:34:18.994 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-08-06T17:34:19.025 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-08-06T17:34:19.026 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi107.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive'} 2024-08-06T17:34:19.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-06T17:34:19.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:34:19.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-06T17:34:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:34:19.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-08-06T17:34:19.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T17:34:19.111 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-08-06T17:34:19.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-08-06T17:34:19.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2024-08-06T17:34:19.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-08-06T17:34:19.203 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-08-06T17:34:19.213 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-08-06T17:34:19.215 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-08-06T17:34:19.347 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-08-06T17:34:19.359 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-08-06T17:34:19.359 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi107.front.sepia.ceph.com,smithi117.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-08-06T17:38:57.767 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi107.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com')] 2024-08-06T17:38:57.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:38:57.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:38:57.857 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-08-06T17:38:57.938 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T17:38:57.938 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:38:57.939 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:38:58.030 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-08-06T17:38:58.118 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T17:38:58.119 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:38:58.120 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T17:38:58.201 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-08-06T17:38:58.284 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T17:38:58.285 INFO:teuthology.run_tasks:Running task clock... 2024-08-06T17:38:58.297 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-08-06T17:38:58.297 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-06T17:38:58.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T17:38:58.300 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-06T17:38:58.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T17:38:58.302 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-08-06T17:38:58.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T17:38:58.328 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-06T17:38:58.330 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-06T17:38:58.342 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-06T17:38:58.344 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-06T17:38:58.352 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-08-06T17:38:58.366 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-08-06T17:38:58.367 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-08-06T17:38:58.369 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2024-08-06T17:38:58.376 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-08-06T17:38:58.380 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2024-08-06T17:38:58.390 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-06T17:38:58.391 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-08-06T17:38:58.395 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-06T17:38:58.402 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-08-06T17:38:58.404 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-06T17:38:58.408 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-06T17:38:58.416 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-08-06T17:38:58.430 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-08-06T17:38:58.458 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-08-06T17:38:58.461 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T17:38:58.462 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-08-06T17:38:58.462 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.462 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.462 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.462 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.464 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.468 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.484 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-08-06T17:38:58.488 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T17:38:58.488 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-08-06T17:38:58.488 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.489 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.489 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.489 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T17:38:58.490 INFO:teuthology.run_tasks:Running task install... 2024-08-06T17:38:58.503 DEBUG:teuthology.task.install:project ceph 2024-08-06T17:38:58.503 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1'}} 2024-08-06T17:38:58.503 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'flavor': 'default', 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1'} 2024-08-06T17:38:58.503 INFO:teuthology.task.install:Using flavor: default 2024-08-06T17:38:58.510 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-08-06T17:38:58.511 INFO:teuthology.task.install:extra packages: [] 2024-08-06T17:38:58.511 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1', 'tag': None, 'wait_for_package': False} 2024-08-06T17:38:58.511 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.512 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.512 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.512 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.512 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.512 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:38:58.514 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1', 'tag': None, 'wait_for_package': False} 2024-08-06T17:38:58.515 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.515 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.515 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.515 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.515 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.515 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:38:58.516 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo', 'ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1', 'tag': None, 'wait_for_package': False} 2024-08-06T17:38:58.516 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.516 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.516 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.516 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.516 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.516 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:38:58.743 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/ 2024-08-06T17:38:58.744 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1236.gb2513712 2024-08-06T17:38:58.778 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/ 2024-08-06T17:38:58.778 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1236.gb2513712 2024-08-06T17:38:58.798 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/ 2024-08-06T17:38:58.799 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1236.gb2513712 2024-08-06T17:38:58.886 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-06T17:38:58.887 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:38:58.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-06T17:38:58.911 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-06T17:38:58.912 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:38:58.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-06T17:38:58.913 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-06T17:38:58.913 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.913 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.913 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.914 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.914 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.914 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-06T17:38:58.938 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/quincy/b251371294e240a32cffa47e4239b888a7835076/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-08-06T17:38:58.939 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:38:58.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-08-06T17:38:58.940 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-06T17:38:58.940 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.940 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.940 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.940 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.940 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.940 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-06T17:38:58.967 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-08-06T17:38:58.967 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:38:58.967 INFO:teuthology.packaging:ref: None 2024-08-06T17:38:58.967 INFO:teuthology.packaging:tag: None 2024-08-06T17:38:58.967 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:38:58.967 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:38:58.968 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-08-06T17:38:58.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-06T17:38:59.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-06T17:38:59.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-08-06T17:38:59.055 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-06T17:38:59.081 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-06T17:38:59.108 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-08-06T17:38:59.119 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-08-06T17:38:59.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-08-06T17:38:59.144 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2024-08-06T17:38:59.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2024-08-06T17:38:59.172 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-08-06T17:38:59.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-08-06T17:38:59.346 INFO:teuthology.orchestra.run.smithi086.stdout:45 files removed 2024-08-06T17:38:59.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-08-06T17:38:59.381 INFO:teuthology.orchestra.run.smithi107.stdout:45 files removed 2024-08-06T17:38:59.404 INFO:teuthology.orchestra.run.smithi117.stdout:45 files removed 2024-08-06T17:38:59.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2024-08-06T17:38:59.430 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw 2024-08-06T17:38:59.981 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 234 kB/s | 77 kB 00:00 2024-08-06T17:39:00.015 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 235 kB/s | 77 kB 00:00 2024-08-06T17:39:00.025 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 243 kB/s | 77 kB 00:00 2024-08-06T17:39:00.440 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 26 kB/s | 11 kB 00:00 2024-08-06T17:39:00.490 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-08-06T17:39:00.501 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 25 kB/s | 11 kB 00:00 2024-08-06T17:39:00.807 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 5.5 kB/s | 1.8 kB 00:00 2024-08-06T17:39:00.824 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 6.2 kB/s | 1.8 kB 00:00 2024-08-06T17:39:00.843 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 6.0 kB/s | 1.8 kB 00:00 2024-08-06T17:39:02.741 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-08-06T17:39:02.760 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.2 MB 00:01 2024-08-06T17:39:02.990 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 3.8 MB/s | 8.2 MB 00:02 2024-08-06T17:39:04.933 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 23 MB/s | 20 MB 00:00 2024-08-06T17:39:05.692 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - AppStream 11 MB/s | 20 MB 00:01 2024-08-06T17:39:06.327 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 20 MB 00:02 2024-08-06T17:39:10.328 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 4.0 MB/s | 6.6 MB 00:01 2024-08-06T17:39:11.967 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - CRB 2.6 MB/s | 6.6 MB 00:02 2024-08-06T17:39:12.361 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 18 kB 00:00 2024-08-06T17:39:13.264 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 29 MB/s | 22 MB 00:00 2024-08-06T17:39:13.277 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.6 MB 00:03 2024-08-06T17:39:13.459 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS Stream 9 - Extras packages 69 kB/s | 18 kB 00:00 2024-08-06T17:39:14.322 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 30 MB/s | 22 MB 00:00 2024-08-06T17:39:15.121 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 18 kB 00:00 2024-08-06T17:39:16.082 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 27 MB/s | 22 MB 00:00 2024-08-06T17:39:18.612 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 50 kB/s | 1.7 kB 00:00 2024-08-06T17:39:19.669 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 35 kB/s | 1.7 kB 00:00 2024-08-06T17:39:20.983 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:39:20.995 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:20.995 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-08-06T17:39:20.995 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:20.995 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:39:20.995 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 14 M 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.1 M 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:17.2.7-1236.gb2513712.el9 ceph 5.0 M 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 22 M 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 636 k 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-06T17:39:20.996 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 457 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 4.9 M 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 45 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 120 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:17.2.7-1236.gb2513712.el9 ceph 181 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-06T17:39:20.997 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:17.2.7-1236.gb2513712.el9 ceph 369 k 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 342 k 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 116 k 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout:Install 22 Packages 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-08-06T17:39:20.998 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:20.999 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-08-06T17:39:20.999 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:39:21.646 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 23 kB/s | 1.7 kB 00:00 2024-08-06T17:39:21.788 INFO:teuthology.orchestra.run.smithi086.stdout:(1/24): ceph-base-17.2.7-1236.gb2513712.el9.x86 11 MB/s | 5.0 MB 00:00 2024-08-06T17:39:21.838 INFO:teuthology.orchestra.run.smithi086.stdout:(2/24): ceph-selinux-17.2.7-1236.gb2513712.el9. 507 kB/s | 25 kB 00:00 2024-08-06T17:39:21.914 INFO:teuthology.orchestra.run.smithi086.stdout:(3/24): libcephfs2-17.2.7-1236.gb2513712.el9.x8 8.3 MB/s | 636 kB 00:00 2024-08-06T17:39:21.989 INFO:teuthology.orchestra.run.smithi086.stdout:(4/24): libradosstriper1-17.2.7-1236.gb2513712. 6.0 MB/s | 457 kB 00:00 2024-08-06T17:39:22.027 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:39:22.037 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 14 M 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.1 M 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:17.2.7-1236.gb2513712.el9 ceph 5.0 M 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 22 M 2024-08-06T17:39:22.038 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 636 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 457 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 4.9 M 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-06T17:39:22.039 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 45 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 120 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:17.2.7-1236.gb2513712.el9 ceph 181 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:17.2.7-1236.gb2513712.el9 ceph 369 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 342 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 116 k 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:22.040 INFO:teuthology.orchestra.run.smithi107.stdout:Install 22 Packages 2024-08-06T17:39:22.041 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2024-08-06T17:39:22.041 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:22.041 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 58 M 2024-08-06T17:39:22.041 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:39:22.281 INFO:teuthology.orchestra.run.smithi086.stdout:(5/24): librgw2-17.2.7-1236.gb2513712.el9.x86_6 17 MB/s | 4.9 MB 00:00 2024-08-06T17:39:22.348 INFO:teuthology.orchestra.run.smithi086.stdout:(6/24): python3-ceph-argparse-17.2.7-1236.gb251 678 kB/s | 45 kB 00:00 2024-08-06T17:39:22.423 INFO:teuthology.orchestra.run.smithi086.stdout:(7/24): python3-ceph-common-17.2.7-1236.gb25137 1.6 MB/s | 120 kB 00:00 2024-08-06T17:39:22.624 INFO:teuthology.orchestra.run.smithi086.stdout:(8/24): ceph-radosgw-17.2.7-1236.gb2513712.el9. 11 MB/s | 14 MB 00:01 2024-08-06T17:39:22.648 INFO:teuthology.orchestra.run.smithi086.stdout:(9/24): python3-cephfs-17.2.7-1236.gb2513712.el 804 kB/s | 181 kB 00:00 2024-08-06T17:39:22.857 INFO:teuthology.orchestra.run.smithi086.stdout:(10/24): ceph-common-17.2.7-1236.gb2513712.el9. 15 MB/s | 22 MB 00:01 2024-08-06T17:39:22.883 INFO:teuthology.orchestra.run.smithi086.stdout:(11/24): python3-rbd-17.2.7-1236.gb2513712.el9. 1.4 MB/s | 342 kB 00:00 2024-08-06T17:39:22.916 INFO:teuthology.orchestra.run.smithi086.stdout:(12/24): python3-rados-17.2.7-1236.gb2513712.el 1.2 MB/s | 369 kB 00:00 2024-08-06T17:39:22.933 INFO:teuthology.orchestra.run.smithi086.stdout:(13/24): python3-rgw-17.2.7-1236.gb2513712.el9. 1.5 MB/s | 116 kB 00:00 2024-08-06T17:39:22.970 INFO:teuthology.orchestra.run.smithi107.stdout:(1/24): ceph-radosgw-17.2.7-1236.gb2513712.el9. 18 MB/s | 14 MB 00:00 2024-08-06T17:39:23.011 INFO:teuthology.orchestra.run.smithi107.stdout:(2/24): ceph-selinux-17.2.7-1236.gb2513712.el9. 609 kB/s | 25 kB 00:00 2024-08-06T17:39:23.058 INFO:teuthology.orchestra.run.smithi086.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 319 kB/s | 45 kB 00:00 2024-08-06T17:39:23.062 INFO:teuthology.orchestra.run.smithi107.stdout:(3/24): libcephfs2-17.2.7-1236.gb2513712.el9.x8 12 MB/s | 636 kB 00:00 2024-08-06T17:39:23.084 INFO:teuthology.orchestra.run.smithi086.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 799 kB/s | 160 kB 00:00 2024-08-06T17:39:23.112 INFO:teuthology.orchestra.run.smithi107.stdout:(4/24): libradosstriper1-17.2.7-1236.gb2513712. 9.0 MB/s | 457 kB 00:00 2024-08-06T17:39:23.134 INFO:teuthology.orchestra.run.smithi086.stdout:(16/24): python3-prettytable-0.7.2-27.el9.noarc 846 kB/s | 42 kB 00:00 2024-08-06T17:39:23.201 INFO:teuthology.orchestra.run.smithi086.stdout:(17/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-08-06T17:39:23.226 INFO:teuthology.orchestra.run.smithi086.stdout:(18/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 292 kB 00:00 2024-08-06T17:39:23.304 INFO:teuthology.orchestra.run.smithi107.stdout:(5/24): librgw2-17.2.7-1236.gb2513712.el9.x86_6 26 MB/s | 4.9 MB 00:00 2024-08-06T17:39:23.309 INFO:teuthology.orchestra.run.smithi086.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.7 MB/s | 309 kB 00:00 2024-08-06T17:39:23.335 INFO:teuthology.orchestra.run.smithi086.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 369 kB/s | 49 kB 00:00 2024-08-06T17:39:23.354 INFO:teuthology.orchestra.run.smithi107.stdout:(6/24): python3-ceph-argparse-17.2.7-1236.gb251 908 kB/s | 45 kB 00:00 2024-08-06T17:39:23.360 INFO:teuthology.orchestra.run.smithi086.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 623 kB/s | 67 kB 00:00 2024-08-06T17:39:23.410 INFO:teuthology.orchestra.run.smithi086.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-08-06T17:39:23.422 INFO:teuthology.orchestra.run.smithi107.stdout:(7/24): ceph-base-17.2.7-1236.gb2513712.el9.x86 4.0 MB/s | 5.0 MB 00:01 2024-08-06T17:39:23.438 INFO:teuthology.orchestra.run.smithi107.stdout:(8/24): python3-ceph-common-17.2.7-1236.gb25137 1.4 MB/s | 120 kB 00:00 2024-08-06T17:39:23.489 INFO:teuthology.orchestra.run.smithi107.stdout:(9/24): python3-cephfs-17.2.7-1236.gb2513712.el 2.6 MB/s | 181 kB 00:00 2024-08-06T17:39:23.502 INFO:teuthology.orchestra.run.smithi086.stdout:(23/24): librados2-17.2.7-1236.gb2513712.el9.x8 18 MB/s | 3.1 MB 00:00 2024-08-06T17:39:23.514 INFO:teuthology.orchestra.run.smithi107.stdout:(10/24): python3-rados-17.2.7-1236.gb2513712.el 4.8 MB/s | 369 kB 00:00 2024-08-06T17:39:23.556 INFO:teuthology.orchestra.run.smithi107.stdout:(11/24): python3-rbd-17.2.7-1236.gb2513712.el9. 5.0 MB/s | 342 kB 00:00 2024-08-06T17:39:23.581 INFO:teuthology.orchestra.run.smithi107.stdout:(12/24): python3-rgw-17.2.7-1236.gb2513712.el9. 1.7 MB/s | 116 kB 00:00 2024-08-06T17:39:23.603 INFO:teuthology.orchestra.run.smithi086.stdout:(24/24): librbd1-17.2.7-1236.gb2513712.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-08-06T17:39:23.607 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:23.607 INFO:teuthology.orchestra.run.smithi086.stdout:Total 22 MB/s | 58 MB 00:02 2024-08-06T17:39:23.668 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:39:23.673 INFO:teuthology.orchestra.run.smithi107.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 494 kB/s | 45 kB 00:00 2024-08-06T17:39:23.700 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:39:23.700 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:39:23.715 INFO:teuthology.orchestra.run.smithi107.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 160 kB 00:00 2024-08-06T17:39:23.774 INFO:teuthology.orchestra.run.smithi107.stdout:(15/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.9 MB/s | 292 kB 00:00 2024-08-06T17:39:23.799 INFO:teuthology.orchestra.run.smithi107.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 5.1 MB/s | 662 kB 00:00 2024-08-06T17:39:23.825 INFO:teuthology.orchestra.run.smithi107.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 842 kB/s | 42 kB 00:00 2024-08-06T17:39:23.911 INFO:teuthology.orchestra.run.smithi107.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 2.8 MB/s | 309 kB 00:00 2024-08-06T17:39:23.934 INFO:teuthology.orchestra.run.smithi107.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.9 MB/s | 67 kB 00:00 2024-08-06T17:39:23.967 INFO:teuthology.orchestra.run.smithi107.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 342 kB/s | 49 kB 00:00 2024-08-06T17:39:23.968 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 14 M 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-08-06T17:39:23.972 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.1 M 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:17.2.7-1236.gb2513712.el9 ceph 5.0 M 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 22 M 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 636 k 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-08-06T17:39:23.973 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 457 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:17.2.7-1236.gb2513712.el9 ceph 4.9 M 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 45 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:17.2.7-1236.gb2513712.el9 ceph 120 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:17.2.7-1236.gb2513712.el9 ceph 181 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-08-06T17:39:23.974 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:17.2.7-1236.gb2513712.el9 ceph 369 k 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 342 k 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:17.2.7-1236.gb2513712.el9 ceph 116 k 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout:Install 22 Packages 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 2 Packages 2024-08-06T17:39:23.975 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:23.976 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 58 M 2024-08-06T17:39:23.976 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:39:24.017 INFO:teuthology.orchestra.run.smithi107.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-08-06T17:39:24.168 INFO:teuthology.orchestra.run.smithi107.stdout:(22/24): librbd1-17.2.7-1236.gb2513712.el9.x86_ 20 MB/s | 2.9 MB 00:00 2024-08-06T17:39:24.244 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:39:24.244 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:39:24.368 INFO:teuthology.orchestra.run.smithi107.stdout:(23/24): librados2-17.2.7-1236.gb2513712.el9.x8 7.7 MB/s | 3.1 MB 00:00 2024-08-06T17:39:24.618 INFO:teuthology.orchestra.run.smithi107.stdout:(24/24): ceph-common-17.2.7-1236.gb2513712.el9. 9.2 MB/s | 22 MB 00:02 2024-08-06T17:39:24.622 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:24.623 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 58 MB 00:02 2024-08-06T17:39:24.684 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:39:24.717 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:39:24.717 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:39:24.997 INFO:teuthology.orchestra.run.smithi117.stdout:(1/24): ceph-base-17.2.7-1236.gb2513712.el9.x86 11 MB/s | 5.0 MB 00:00 2024-08-06T17:39:25.056 INFO:teuthology.orchestra.run.smithi117.stdout:(2/24): ceph-selinux-17.2.7-1236.gb2513712.el9. 433 kB/s | 25 kB 00:00 2024-08-06T17:39:25.139 INFO:teuthology.orchestra.run.smithi117.stdout:(3/24): libcephfs2-17.2.7-1236.gb2513712.el9.x8 7.5 MB/s | 636 kB 00:00 2024-08-06T17:39:25.215 INFO:teuthology.orchestra.run.smithi117.stdout:(4/24): libradosstriper1-17.2.7-1236.gb2513712. 6.0 MB/s | 457 kB 00:00 2024-08-06T17:39:25.254 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:39:25.254 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:39:25.398 INFO:teuthology.orchestra.run.smithi117.stdout:(5/24): ceph-radosgw-17.2.7-1236.gb2513712.el9. 16 MB/s | 14 MB 00:00 2024-08-06T17:39:25.430 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:39:25.449 INFO:teuthology.orchestra.run.smithi117.stdout:(6/24): python3-ceph-argparse-17.2.7-1236.gb251 899 kB/s | 45 kB 00:00 2024-08-06T17:39:25.515 INFO:teuthology.orchestra.run.smithi117.stdout:(7/24): librgw2-17.2.7-1236.gb2513712.el9.x86_6 16 MB/s | 4.9 MB 00:00 2024-08-06T17:39:25.541 INFO:teuthology.orchestra.run.smithi117.stdout:(8/24): python3-ceph-common-17.2.7-1236.gb25137 1.3 MB/s | 120 kB 00:00 2024-08-06T17:39:25.566 INFO:teuthology.orchestra.run.smithi117.stdout:(9/24): python3-cephfs-17.2.7-1236.gb2513712.el 3.5 MB/s | 181 kB 00:00 2024-08-06T17:39:25.600 INFO:teuthology.orchestra.run.smithi117.stdout:(10/24): python3-rados-17.2.7-1236.gb2513712.el 6.1 MB/s | 369 kB 00:00 2024-08-06T17:39:25.625 INFO:teuthology.orchestra.run.smithi117.stdout:(11/24): python3-rbd-17.2.7-1236.gb2513712.el9. 5.7 MB/s | 342 kB 00:00 2024-08-06T17:39:25.650 INFO:teuthology.orchestra.run.smithi117.stdout:(12/24): python3-rgw-17.2.7-1236.gb2513712.el9. 2.3 MB/s | 116 kB 00:00 2024-08-06T17:39:25.803 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-06T17:39:25.855 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:25.950 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:25.959 INFO:teuthology.orchestra.run.smithi117.stdout:(13/24): ceph-common-17.2.7-1236.gb2513712.el9. 16 MB/s | 22 MB 00:01 2024-08-06T17:39:26.025 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:26.065 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:26.150 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:26.196 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-06T17:39:26.228 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-06T17:39:26.359 INFO:teuthology.orchestra.run.smithi117.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 64 kB/s | 45 kB 00:00 2024-08-06T17:39:26.430 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:39:26.442 INFO:teuthology.orchestra.run.smithi117.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 196 kB/s | 160 kB 00:00 2024-08-06T17:39:26.568 INFO:teuthology.orchestra.run.smithi117.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.4 MB/s | 292 kB 00:00 2024-08-06T17:39:26.584 INFO:teuthology.orchestra.run.smithi117.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 298 kB/s | 42 kB 00:00 2024-08-06T17:39:26.658 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-06T17:39:26.668 INFO:teuthology.orchestra.run.smithi117.stdout:(18/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 3.0 MB/s | 309 kB 00:00 2024-08-06T17:39:26.684 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:26.685 INFO:teuthology.orchestra.run.smithi117.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 4.7 MB/s | 67 kB 00:00 2024-08-06T17:39:26.702 INFO:teuthology.orchestra.run.smithi117.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 414 kB/s | 49 kB 00:00 2024-08-06T17:39:26.747 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:26.761 INFO:teuthology.orchestra.run.smithi117.stdout:(21/24): librdkafka-1.6.1-102.el9.x86_64.rpm 880 kB/s | 662 kB 00:00 2024-08-06T17:39:26.775 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-06T17:39:26.802 INFO:teuthology.orchestra.run.smithi117.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-08-06T17:39:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-06T17:39:26.821 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 10/26 2024-08-06T17:39:26.868 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:26.870 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:26.934 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:26.936 INFO:teuthology.orchestra.run.smithi117.stdout:(23/24): librados2-17.2.7-1236.gb2513712.el9.x8 13 MB/s | 3.1 MB 00:00 2024-08-06T17:39:26.949 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:26.962 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:27.003 INFO:teuthology.orchestra.run.smithi117.stdout:(24/24): librbd1-17.2.7-1236.gb2513712.el9.x86_ 12 MB/s | 2.9 MB 00:00 2024-08-06T17:39:27.009 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:27.009 INFO:teuthology.orchestra.run.smithi117.stdout:Total 19 MB/s | 58 MB 00:03 2024-08-06T17:39:27.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:27.045 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:27.069 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:39:27.072 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:27.085 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-06T17:39:27.101 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:39:27.101 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:39:27.129 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-06T17:39:27.146 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:27.191 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-06T17:39:27.195 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-06T17:39:27.227 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-06T17:39:27.480 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-06T17:39:27.533 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:27.609 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:27.657 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:39:27.658 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:39:27.664 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-06T17:39:27.680 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 19/26 2024-08-06T17:39:27.692 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:27.710 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 20/26 2024-08-06T17:39:27.754 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:27.783 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-06T17:39:27.828 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 10/26 2024-08-06T17:39:27.867 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:27.933 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:27.961 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:28.010 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:28.051 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-06T17:39:28.086 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-06T17:39:28.119 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-06T17:39:28.412 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-06T17:39:28.457 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:28.551 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:28.620 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 19/26 2024-08-06T17:39:28.651 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 20/26 2024-08-06T17:39:28.828 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:39:29.224 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-08-06T17:39:29.272 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:29.368 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:29.469 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:29.498 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:29.592 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:29.632 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-08-06T17:39:29.670 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-08-06T17:39:30.102 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-08-06T17:39:30.126 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:30.143 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:30.180 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 8/26 2024-08-06T17:39:30.182 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:30.209 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-08-06T17:39:30.254 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 10/26 2024-08-06T17:39:30.291 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:30.350 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:30.379 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:30.445 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 13/26 2024-08-06T17:39:30.514 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-08-06T17:39:30.546 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-08-06T17:39:30.578 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-08-06T17:39:30.721 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:30.761 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:30.848 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-06T17:39:30.848 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-06T17:39:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:30.870 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-08-06T17:39:30.879 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:30.908 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:31.001 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 18/26 2024-08-06T17:39:31.007 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:31.048 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:31.071 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 19/26 2024-08-06T17:39:31.102 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 20/26 2024-08-06T17:39:31.581 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:31.618 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:31.716 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:31.717 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-06T17:39:31.717 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-06T17:39:31.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:31.753 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:33.520 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:33.557 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:34.110 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 21/26 2024-08-06T17:39:34.144 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:34.241 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 22/26 2024-08-06T17:39:34.242 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-08-06T17:39:34.242 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-08-06T17:39:34.242 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:34.287 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:40.536 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:40.536 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-08-06T17:39:40.536 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-08-06T17:39:40.536 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-08-06T17:39:40.536 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-08-06T17:39:40.537 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-08-06T17:39:40.537 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-08-06T17:39:40.537 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-08-06T17:39:40.537 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2024-08-06T17:39:41.382 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2024-08-06T17:39:41.383 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /root 2024-08-06T17:39:41.383 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2024-08-06T17:39:41.383 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:41.840 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:41.918 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:41.920 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:42.018 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:42.655 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:42.733 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:42.733 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:39:42.733 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-06T17:39:42.734 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:42.734 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:42.734 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:42.735 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:42.816 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:42.817 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 1/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 5/26 2024-08-06T17:39:43.196 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 6/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 7/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 8/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 9/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 10/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 13/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-06T17:39:43.197 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-06T17:39:43.198 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 25/26 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-08-06T17:39:43.803 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 1/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 5/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 6/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 7/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 8/26 2024-08-06T17:39:44.007 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 9/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 10/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 13/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-06T17:39:44.008 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-06T17:39:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-06T17:39:44.009 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 25/26 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:39:44.092 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-06T17:39:44.093 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:44.094 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:39:44.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-08-06T17:39:44.636 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-06T17:39:44.637 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.638 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:44.639 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:39:45.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2024-08-06T17:39:45.011 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:27 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:39:45.072 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:45.136 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:17.2.7-1236.gb2513712.el9 ceph 48 M 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:17.2.7-1236.gb2513712.el9 ceph 163 k 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:45.137 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 192 M 2024-08-06T17:39:45.138 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 24/26 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-08-06T17:39:45.151 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:45.153 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:45.242 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-08-06T17:39:45.242 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:45.440 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): libcephsqlite-17.2.7-1236.gb2513712.el9. 684 kB/s | 163 kB 00:00 2024-08-06T17:39:45.465 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.1 MB/s | 304 kB 00:00 2024-08-06T17:39:45.569 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:26 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:39:45.608 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 380 kB/s | 64 kB 00:00 2024-08-06T17:39:45.696 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:17.2.7-1236.gb2513712.el9 ceph 48 M 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite x86_64 2:17.2.7-1236.gb2513712.el9 ceph 163 k 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:39:45.697 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:45.698 INFO:teuthology.orchestra.run.smithi107.stdout:Install 4 Packages 2024-08-06T17:39:45.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:45.698 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 48 M 2024-08-06T17:39:45.698 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 192 M 2024-08-06T17:39:45.698 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:39:46.046 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.3 MB/s | 304 kB 00:00 2024-08-06T17:39:46.096 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.3 MB/s | 64 kB 00:00 2024-08-06T17:39:46.254 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): libcephsqlite-17.2.7-1236.gb2513712.el9. 369 kB/s | 163 kB 00:00 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 1/26 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2/26 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 3/26 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 4/26 2024-08-06T17:39:46.508 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 5/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_6 6/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 7/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9. 8/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x8 9/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 10/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 11/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 12/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 13/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-08-06T17:39:46.509 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:17.2.7-1236.gb2513712.el9.x86_64 23/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-08-06T17:39:46.510 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 25/26 2024-08-06T17:39:46.732 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): ceph-test-17.2.7-1236.gb2513712.el9.x86_ 31 MB/s | 48 MB 00:01 2024-08-06T17:39:46.734 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:46.735 INFO:teuthology.orchestra.run.smithi086.stdout:Total 30 MB/s | 48 MB 00:01 2024-08-06T17:39:46.761 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:39:46.780 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:39:46.781 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:39:47.144 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:39:47.144 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.318 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-08-06T17:39:47.319 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:47.320 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:39:47.661 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:39:47.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-test 2024-08-06T17:39:47.751 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-06T17:39:47.798 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-06T17:39:47.826 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:48.154 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): ceph-test-17.2.7-1236.gb2513712.el9.x86_ 20 MB/s | 48 MB 00:02 2024-08-06T17:39:48.156 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:48.156 INFO:teuthology.orchestra.run.smithi107.stdout:Total 20 MB/s | 48 MB 00:02 2024-08-06T17:39:48.183 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:39:48.202 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:39:48.203 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:39:48.253 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:27 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:39:48.378 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:39:48.378 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:17.2.7-1236.gb2513712.el9 ceph 48 M 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:17.2.7-1236.gb2513712.el9 ceph 163 k 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:48.379 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout:Install 4 Packages 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 48 M 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 192 M 2024-08-06T17:39:48.380 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:39:48.567 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:39:48.567 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:39:48.898 INFO:teuthology.orchestra.run.smithi117.stdout:(1/4): libcephsqlite-17.2.7-1236.gb2513712.el9. 667 kB/s | 163 kB 00:00 2024-08-06T17:39:49.092 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:39:49.182 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-06T17:39:49.228 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-06T17:39:49.248 INFO:teuthology.orchestra.run.smithi117.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 512 kB/s | 304 kB 00:00 2024-08-06T17:39:49.256 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:49.273 INFO:teuthology.orchestra.run.smithi117.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 170 kB/s | 64 kB 00:00 2024-08-06T17:39:50.149 INFO:teuthology.orchestra.run.smithi117.stdout:(4/4): ceph-test-17.2.7-1236.gb2513712.el9.x86_ 32 MB/s | 48 MB 00:01 2024-08-06T17:39:50.150 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:50.151 INFO:teuthology.orchestra.run.smithi117.stdout:Total 27 MB/s | 48 MB 00:01 2024-08-06T17:39:50.178 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:39:50.198 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:39:50.198 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:39:50.561 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:39:50.561 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:39:51.091 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:39:51.173 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-08-06T17:39:51.222 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-08-06T17:39:51.250 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:52.205 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:52.274 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:53.659 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:53.730 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:54.519 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:54.519 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 1/4 2024-08-06T17:39:54.519 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2/4 2024-08-06T17:39:54.519 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-06T17:39:54.799 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:54.800 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:39:55.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-08-06T17:39:55.643 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 3/4 2024-08-06T17:39:55.646 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:37 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:39:55.714 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:55.775 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:17.2.7-1236.gb2513712.el9 ceph 6.5 k 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.1 M 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:17.2.7-1236.gb2513712.el9 ceph 1.4 M 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 240 k 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.9 M 2024-08-06T17:39:55.778 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 15 M 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-06T17:39:55.779 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-06T17:39:55.780 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout:Install 34 Packages 2024-08-06T17:39:55.781 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:39:55.782 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 27 M 2024-08-06T17:39:55.782 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 96 M 2024-08-06T17:39:55.782 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:39:56.125 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:56.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 1/4 2024-08-06T17:39:56.125 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2/4 2024-08-06T17:39:56.126 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:56.394 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:39:56.417 INFO:teuthology.orchestra.run.smithi086.stdout:(1/34): ceph-17.2.7-1236.gb2513712.el9.x86_64.r 40 kB/s | 6.5 kB 00:00 2024-08-06T17:39:56.592 INFO:teuthology.orchestra.run.smithi086.stdout:(2/34): ceph-mgr-17.2.7-1236.gb2513712.el9.x86_ 4.1 MB/s | 1.4 MB 00:00 2024-08-06T17:39:56.634 INFO:teuthology.orchestra.run.smithi086.stdout:(3/34): ceph-mds-17.2.7-1236.gb2513712.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-08-06T17:39:56.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2024-08-06T17:39:56.717 INFO:teuthology.orchestra.run.smithi086.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1236.gb251 2.8 MB/s | 240 kB 00:00 2024-08-06T17:39:56.818 INFO:teuthology.orchestra.run.smithi086.stdout:(5/34): ceph-mon-17.2.7-1236.gb2513712.el9.x86_ 9.8 MB/s | 3.9 MB 00:00 2024-08-06T17:39:57.110 INFO:teuthology.orchestra.run.smithi086.stdout:(6/34): ceph-osd-17.2.7-1236.gb2513712.el9.x86_ 30 MB/s | 15 MB 00:00 2024-08-06T17:39:57.239 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:38 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:39:57.371 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:39:57.373 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:17.2.7-1236.gb2513712.el9 ceph 6.5 k 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.1 M 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:17.2.7-1236.gb2513712.el9 ceph 1.4 M 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 240 k 2024-08-06T17:39:57.374 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.9 M 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 15 M 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-06T17:39:57.375 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-06T17:39:57.376 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:39:57.377 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:39:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:Install 34 Packages 2024-08-06T17:39:57.378 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:39:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 27 M 2024-08-06T17:39:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 96 M 2024-08-06T17:39:57.378 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:39:57.535 INFO:teuthology.orchestra.run.smithi086.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 310 kB/s | 253 kB 00:00 2024-08-06T17:39:57.577 INFO:teuthology.orchestra.run.smithi086.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 230 kB/s | 106 kB 00:00 2024-08-06T17:39:57.660 INFO:teuthology.orchestra.run.smithi086.stdout:(9/34): python3-pycparser-2.20-6.el9.noarch.rpm 1.1 MB/s | 135 kB 00:00 2024-08-06T17:39:57.702 INFO:teuthology.orchestra.run.smithi086.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 997 kB/s | 125 kB 00:00 2024-08-06T17:39:57.736 INFO:teuthology.orchestra.run.smithi086.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 1.3 MB/s | 1.2 MB 00:00 2024-08-06T17:39:57.795 INFO:teuthology.orchestra.run.smithi086.stdout:(12/34): python3-urllib3-1.26.5-6.el9.noarch.rp 1.6 MB/s | 215 kB 00:00 2024-08-06T17:39:57.820 INFO:teuthology.orchestra.run.smithi086.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 416 kB/s | 35 kB 00:00 2024-08-06T17:39:57.855 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 4/4 2024-08-06T17:39:57.855 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 1/4 2024-08-06T17:39:57.855 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2/4 2024-08-06T17:39:57.855 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-08-06T17:39:57.870 INFO:teuthology.orchestra.run.smithi086.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.0 MB/s | 172 kB 00:00 2024-08-06T17:39:57.895 INFO:teuthology.orchestra.run.smithi086.stdout:(15/34): python3-pytz-2021.1-5.el9.noarch.rpm 677 kB/s | 51 kB 00:00 2024-08-06T17:39:57.938 INFO:teuthology.orchestra.run.smithi086.stdout:(16/34): python3-packaging-20.9-5.el9.noarch.rp 541 kB/s | 77 kB 00:00 2024-08-06T17:39:57.956 INFO:teuthology.orchestra.run.smithi107.stdout:(1/34): ceph-17.2.7-1236.gb2513712.el9.x86_64.r 39 kB/s | 6.5 kB 00:00 2024-08-06T17:39:57.963 INFO:teuthology.orchestra.run.smithi086.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 452 kB/s | 42 kB 00:00 2024-08-06T17:39:58.071 INFO:teuthology.orchestra.run.smithi086.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 247 kB/s | 43 kB 00:00 2024-08-06T17:39:58.097 INFO:teuthology.orchestra.run.smithi086.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 478 kB/s | 11 kB 00:00 2024-08-06T17:39:58.122 INFO:teuthology.orchestra.run.smithi086.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 711 kB/s | 18 kB 00:00 2024-08-06T17:39:58.147 INFO:teuthology.orchestra.run.smithi086.stdout:(21/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 933 kB/s | 172 kB 00:00 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-6.el9.x86_64 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:58.157 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:39:58.173 INFO:teuthology.orchestra.run.smithi086.stdout:(22/34): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-08-06T17:39:58.173 INFO:teuthology.orchestra.run.smithi107.stdout:(2/34): ceph-mds-17.2.7-1236.gb2513712.el9.x86_ 5.4 MB/s | 2.1 MB 00:00 2024-08-06T17:39:58.198 INFO:teuthology.orchestra.run.smithi086.stdout:(23/34): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-08-06T17:39:58.224 INFO:teuthology.orchestra.run.smithi086.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-08-06T17:39:58.224 INFO:teuthology.orchestra.run.smithi107.stdout:(3/34): ceph-mgr-17.2.7-1236.gb2513712.el9.x86_ 3.2 MB/s | 1.4 MB 00:00 2024-08-06T17:39:58.249 INFO:teuthology.orchestra.run.smithi086.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 923 kB/s | 46 kB 00:00 2024-08-06T17:39:58.274 INFO:teuthology.orchestra.run.smithi086.stdout:(26/34): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-08-06T17:39:58.299 INFO:teuthology.orchestra.run.smithi086.stdout:(27/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.0 MB/s | 358 kB 00:00 2024-08-06T17:39:58.307 INFO:teuthology.orchestra.run.smithi107.stdout:(4/34): ceph-mon-17.2.7-1236.gb2513712.el9.x86_ 11 MB/s | 3.9 MB 00:00 2024-08-06T17:39:58.316 INFO:teuthology.orchestra.run.smithi086.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 395 kB/s | 16 kB 00:00 2024-08-06T17:39:58.332 INFO:teuthology.orchestra.run.smithi107.stdout:(5/34): ceph-mgr-modules-core-17.2.7-1236.gb251 2.2 MB/s | 240 kB 00:00 2024-08-06T17:39:58.341 INFO:teuthology.orchestra.run.smithi086.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 1.4 MB/s | 36 kB 00:00 2024-08-06T17:39:58.367 INFO:teuthology.orchestra.run.smithi086.stdout:(30/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-08-06T17:39:58.391 INFO:teuthology.orchestra.run.smithi086.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.9 MB/s | 272 kB 00:00 2024-08-06T17:39:58.408 INFO:teuthology.orchestra.run.smithi086.stdout:(32/34): python3-zc-lockfile-2.0-10.el9.noarch. 1.2 MB/s | 20 kB 00:00 2024-08-06T17:39:58.442 INFO:teuthology.orchestra.run.smithi086.stdout:(33/34): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-08-06T17:39:58.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph 2024-08-06T17:39:58.467 INFO:teuthology.orchestra.run.smithi086.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-06T17:39:58.475 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:39:58.475 INFO:teuthology.orchestra.run.smithi086.stdout:Total 10 MB/s | 27 MB 00:02 2024-08-06T17:39:58.609 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:39:58.647 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:39:58.647 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:39:58.674 INFO:teuthology.orchestra.run.smithi107.stdout:(6/34): ceph-osd-17.2.7-1236.gb2513712.el9.x86_ 31 MB/s | 15 MB 00:00 2024-08-06T17:39:58.699 INFO:teuthology.orchestra.run.smithi107.stdout:(7/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 646 kB/s | 253 kB 00:00 2024-08-06T17:39:58.758 INFO:teuthology.orchestra.run.smithi107.stdout:(8/34): python3-pycparser-2.20-6.el9.noarch.rpm 2.3 MB/s | 135 kB 00:00 2024-08-06T17:39:58.792 INFO:teuthology.orchestra.run.smithi107.stdout:(9/34): python3-cryptography-36.0.1-4.el9.x86_6 2.7 MB/s | 1.2 MB 00:00 2024-08-06T17:39:58.817 INFO:teuthology.orchestra.run.smithi107.stdout:(10/34): python3-requests-2.25.1-8.el9.noarch.r 2.1 MB/s | 125 kB 00:00 2024-08-06T17:39:58.850 INFO:teuthology.orchestra.run.smithi107.stdout:(11/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.6 MB/s | 215 kB 00:00 2024-08-06T17:39:58.876 INFO:teuthology.orchestra.run.smithi107.stdout:(12/34): python3-ply-3.11-14.el9.noarch.rpm 530 kB/s | 106 kB 00:00 2024-08-06T17:39:58.991 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:37 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:39:59.034 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:39:59.034 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:39:59.121 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:17.2.7-1236.gb2513712.el9 ceph 6.5 k 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.1 M 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:17.2.7-1236.gb2513712.el9 ceph 1.4 M 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 240 k 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:17.2.7-1236.gb2513712.el9 ceph 3.9 M 2024-08-06T17:39:59.124 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 15 M 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-08-06T17:39:59.125 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-08-06T17:39:59.126 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout:Install 34 Packages 2024-08-06T17:39:59.127 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:39:59.128 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 27 M 2024-08-06T17:39:59.128 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 96 M 2024-08-06T17:39:59.128 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:39:59.426 INFO:teuthology.orchestra.run.smithi107.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 60 kB/s | 35 kB 00:00 2024-08-06T17:39:59.534 INFO:teuthology.orchestra.run.smithi107.stdout:(14/34): python3-packaging-20.9-5.el9.noarch.rp 117 kB/s | 77 kB 00:00 2024-08-06T17:39:59.626 INFO:teuthology.orchestra.run.smithi107.stdout:(15/34): python3-mako-1.1.4-6.el9.noarch.rpm 213 kB/s | 172 kB 00:00 2024-08-06T17:39:59.710 INFO:teuthology.orchestra.run.smithi107.stdout:(16/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 524 kB/s | 43 kB 00:00 2024-08-06T17:39:59.735 INFO:teuthology.orchestra.run.smithi107.stdout:(17/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 6.8 MB/s | 172 kB 00:00 2024-08-06T17:39:59.769 INFO:teuthology.orchestra.run.smithi107.stdout:(18/34): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-08-06T17:39:59.786 INFO:teuthology.orchestra.run.smithi107.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 674 kB/s | 11 kB 00:00 2024-08-06T17:39:59.828 INFO:teuthology.orchestra.run.smithi107.stdout:(20/34): python3-jaraco-classes-3.2.1-5.el9.noa 1.0 MB/s | 18 kB 00:00 2024-08-06T17:39:59.831 INFO:teuthology.orchestra.run.smithi117.stdout:(1/34): ceph-17.2.7-1236.gb2513712.el9.x86_64.r 33 kB/s | 6.5 kB 00:00 2024-08-06T17:39:59.844 INFO:teuthology.orchestra.run.smithi107.stdout:(21/34): python3-jaraco-collections-3.0.0-8.el9 1.4 MB/s | 23 kB 00:00 2024-08-06T17:39:59.861 INFO:teuthology.orchestra.run.smithi107.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 1.1 MB/s | 19 kB 00:00 2024-08-06T17:39:59.878 INFO:teuthology.orchestra.run.smithi107.stdout:(23/34): python3-jaraco-text-3.2.0-6.el9.noarch 1.2 MB/s | 20 kB 00:00 2024-08-06T17:39:59.895 INFO:teuthology.orchestra.run.smithi107.stdout:(24/34): python3-logutils-0.3.5-21.el9.noarch.r 2.7 MB/s | 46 kB 00:00 2024-08-06T17:39:59.921 INFO:teuthology.orchestra.run.smithi107.stdout:(25/34): python3-more-itertools-8.12.0-2.el9.no 3.2 MB/s | 79 kB 00:00 2024-08-06T17:39:59.946 INFO:teuthology.orchestra.run.smithi107.stdout:(26/34): python3-pecan-1.4.2-3.el9.noarch.rpm 11 MB/s | 272 kB 00:00 2024-08-06T17:39:59.962 INFO:teuthology.orchestra.run.smithi107.stdout:(27/34): python3-portend-3.1.0-2.el9.noarch.rpm 984 kB/s | 16 kB 00:00 2024-08-06T17:39:59.981 INFO:teuthology.orchestra.run.smithi117.stdout:(2/34): ceph-mds-17.2.7-1236.gb2513712.el9.x86_ 5.9 MB/s | 2.1 MB 00:00 2024-08-06T17:39:59.996 INFO:teuthology.orchestra.run.smithi107.stdout:(28/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.6 MB/s | 90 kB 00:00 2024-08-06T17:40:00.013 INFO:teuthology.orchestra.run.smithi107.stdout:(29/34): python3-tempora-5.0.0-2.el9.noarch.rpm 2.1 MB/s | 36 kB 00:00 2024-08-06T17:40:00.015 INFO:teuthology.orchestra.run.smithi117.stdout:(3/34): ceph-mgr-17.2.7-1236.gb2513712.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2024-08-06T17:40:00.038 INFO:teuthology.orchestra.run.smithi107.stdout:(30/34): python3-webob-1.8.7-6.el9.noarch.rpm 8.9 MB/s | 230 kB 00:00 2024-08-06T17:40:00.080 INFO:teuthology.orchestra.run.smithi107.stdout:(31/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 10 MB/s | 427 kB 00:00 2024-08-06T17:40:00.099 INFO:teuthology.orchestra.run.smithi117.stdout:(4/34): ceph-mgr-modules-core-17.2.7-1236.gb251 2.8 MB/s | 240 kB 00:00 2024-08-06T17:40:00.106 INFO:teuthology.orchestra.run.smithi107.stdout:(32/34): python3-pytz-2021.1-5.el9.noarch.rpm 75 kB/s | 51 kB 00:00 2024-08-06T17:40:00.131 INFO:teuthology.orchestra.run.smithi107.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 399 kB/s | 20 kB 00:00 2024-08-06T17:40:00.214 INFO:teuthology.orchestra.run.smithi107.stdout:(34/34): python3-toml-0.10.2-6.el9.noarch.rpm 61 kB/s | 42 kB 00:00 2024-08-06T17:40:00.219 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:00.219 INFO:teuthology.orchestra.run.smithi107.stdout:Total 9.6 MB/s | 27 MB 00:02 2024-08-06T17:40:00.237 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:00.274 INFO:teuthology.orchestra.run.smithi117.stdout:(5/34): ceph-mon-17.2.7-1236.gb2513712.el9.x86_ 8.9 MB/s | 3.9 MB 00:00 2024-08-06T17:40:00.303 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-06T17:40:00.324 INFO:teuthology.orchestra.run.smithi117.stdout:(6/34): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.1 MB/s | 253 kB 00:00 2024-08-06T17:40:00.335 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-06T17:40:00.349 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:00.368 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-06T17:40:00.386 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:00.386 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:00.461 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-06T17:40:00.500 INFO:teuthology.orchestra.run.smithi117.stdout:(7/34): ceph-osd-17.2.7-1236.gb2513712.el9.x86_ 30 MB/s | 15 MB 00:00 2024-08-06T17:40:00.525 INFO:teuthology.orchestra.run.smithi117.stdout:(8/34): python3-ply-3.11-14.el9.noarch.rpm 532 kB/s | 106 kB 00:00 2024-08-06T17:40:00.547 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-06T17:40:00.583 INFO:teuthology.orchestra.run.smithi117.stdout:(9/34): python3-requests-2.25.1-8.el9.noarch.rp 2.1 MB/s | 125 kB 00:00 2024-08-06T17:40:00.588 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-06T17:40:00.623 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-06T17:40:00.650 INFO:teuthology.orchestra.run.smithi117.stdout:(10/34): python3-urllib3-1.26.5-6.el9.noarch.rp 3.2 MB/s | 215 kB 00:00 2024-08-06T17:40:00.653 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-06T17:40:00.684 INFO:teuthology.orchestra.run.smithi117.stdout:(11/34): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.2 MB 00:00 2024-08-06T17:40:00.688 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-06T17:40:00.709 INFO:teuthology.orchestra.run.smithi117.stdout:(12/34): python3-pycparser-2.20-6.el9.noarch.rp 646 kB/s | 135 kB 00:00 2024-08-06T17:40:00.724 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-06T17:40:00.759 INFO:teuthology.orchestra.run.smithi117.stdout:(13/34): python3-markupsafe-1.1.1-12.el9.x86_64 463 kB/s | 35 kB 00:00 2024-08-06T17:40:00.761 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-06T17:40:00.771 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:00.772 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:00.793 INFO:teuthology.orchestra.run.smithi117.stdout:(14/34): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-08-06T17:40:00.796 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-06T17:40:00.818 INFO:teuthology.orchestra.run.smithi117.stdout:(15/34): python3-packaging-20.9-5.el9.noarch.rp 710 kB/s | 77 kB 00:00 2024-08-06T17:40:00.831 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-06T17:40:00.843 INFO:teuthology.orchestra.run.smithi117.stdout:(16/34): python3-pytz-2021.1-5.el9.noarch.rpm 607 kB/s | 51 kB 00:00 2024-08-06T17:40:00.869 INFO:teuthology.orchestra.run.smithi117.stdout:(17/34): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-08-06T17:40:00.879 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-06T17:40:00.911 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:00.984 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-06T17:40:00.994 INFO:teuthology.orchestra.run.smithi117.stdout:(18/34): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 247 kB/s | 43 kB 00:00 2024-08-06T17:40:01.019 INFO:teuthology.orchestra.run.smithi117.stdout:(19/34): python3-jaraco-8.2.1-3.el9.noarch.rpm 471 kB/s | 11 kB 00:00 2024-08-06T17:40:01.044 INFO:teuthology.orchestra.run.smithi117.stdout:(20/34): python3-cheroot-8.6.0-4.el9.noarch.rpm 856 kB/s | 172 kB 00:00 2024-08-06T17:40:01.070 INFO:teuthology.orchestra.run.smithi117.stdout:(21/34): python3-jaraco-classes-3.2.1-5.el9.noa 351 kB/s | 18 kB 00:00 2024-08-06T17:40:01.095 INFO:teuthology.orchestra.run.smithi117.stdout:(22/34): python3-jaraco-functools-3.5.0-2.el9.n 792 kB/s | 19 kB 00:00 2024-08-06T17:40:01.120 INFO:teuthology.orchestra.run.smithi117.stdout:(23/34): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-08-06T17:40:01.122 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-06T17:40:01.146 INFO:teuthology.orchestra.run.smithi117.stdout:(24/34): python3-jaraco-text-3.2.0-6.el9.noarch 388 kB/s | 20 kB 00:00 2024-08-06T17:40:01.171 INFO:teuthology.orchestra.run.smithi117.stdout:(25/34): python3-logutils-0.3.5-21.el9.noarch.r 922 kB/s | 46 kB 00:00 2024-08-06T17:40:01.196 INFO:teuthology.orchestra.run.smithi117.stdout:(26/34): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-08-06T17:40:01.211 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-06T17:40:01.221 INFO:teuthology.orchestra.run.smithi117.stdout:(27/34): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-08-06T17:40:01.247 INFO:teuthology.orchestra.run.smithi117.stdout:(28/34): python3-portend-3.1.0-2.el9.noarch.rpm 326 kB/s | 16 kB 00:00 2024-08-06T17:40:01.264 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-06T17:40:01.272 INFO:teuthology.orchestra.run.smithi117.stdout:(29/34): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-08-06T17:40:01.297 INFO:teuthology.orchestra.run.smithi117.stdout:(30/34): python3-tempora-5.0.0-2.el9.noarch.rpm 716 kB/s | 36 kB 00:00 2024-08-06T17:40:01.306 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-06T17:40:01.322 INFO:teuthology.orchestra.run.smithi117.stdout:(31/34): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-08-06T17:40:01.348 INFO:teuthology.orchestra.run.smithi117.stdout:(32/34): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-08-06T17:40:01.351 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-06T17:40:01.373 INFO:teuthology.orchestra.run.smithi117.stdout:(33/34): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-08-06T17:40:01.398 INFO:teuthology.orchestra.run.smithi117.stdout:(34/34): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-08-06T17:40:01.402 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:01.402 INFO:teuthology.orchestra.run.smithi117.stdout:Total 12 MB/s | 27 MB 00:02 2024-08-06T17:40:01.410 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-06T17:40:01.524 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:01.561 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:01.561 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:01.602 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-06T17:40:01.656 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-06T17:40:01.734 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-06T17:40:01.822 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-06T17:40:01.855 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-06T17:40:01.923 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:01.926 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:01.958 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:01.971 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-06T17:40:02.017 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-06T17:40:02.049 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-06T17:40:02.082 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-06T17:40:02.126 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 29/34 2024-08-06T17:40:02.157 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:02.174 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:02.198 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:02.261 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-06T17:40:02.309 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-06T17:40:02.353 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-06T17:40:02.383 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-06T17:40:02.417 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-06T17:40:02.451 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-06T17:40:02.491 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-06T17:40:02.535 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-06T17:40:02.569 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-06T17:40:02.617 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-06T17:40:02.650 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:02.731 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-06T17:40:02.887 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-06T17:40:02.966 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-06T17:40:03.020 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-06T17:40:03.058 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-06T17:40:03.104 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:03.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-06T17:40:03.165 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-06T17:40:03.175 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/34 2024-08-06T17:40:03.208 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/34 2024-08-06T17:40:03.241 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/34 2024-08-06T17:40:03.332 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/34 2024-08-06T17:40:03.374 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-06T17:40:03.410 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/34 2024-08-06T17:40:03.420 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-06T17:40:03.451 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/34 2024-08-06T17:40:03.486 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/34 2024-08-06T17:40:03.499 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-06T17:40:03.508 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:03.516 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/34 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:03.536 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:03.551 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/34 2024-08-06T17:40:03.588 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-06T17:40:03.595 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/34 2024-08-06T17:40:03.633 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/34 2024-08-06T17:40:03.636 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-06T17:40:03.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/34 2024-08-06T17:40:03.702 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/34 2024-08-06T17:40:03.758 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-06T17:40:03.758 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/34 2024-08-06T17:40:03.783 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:03.856 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 16/34 2024-08-06T17:40:03.900 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 29/34 2024-08-06T17:40:03.912 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:03.930 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:03.936 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-06T17:40:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:03.936 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:03.937 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:03.967 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:03.967 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:03.968 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-06T17:40:03.968 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:03.968 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:03.968 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:04.003 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 17/34 2024-08-06T17:40:04.100 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 18/34 2024-08-06T17:40:04.135 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:04.152 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 19/34 2024-08-06T17:40:04.158 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-06T17:40:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:04.159 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:04.159 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:04.191 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 20/34 2024-08-06T17:40:04.224 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:04.248 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 21/34 2024-08-06T17:40:04.314 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 22/34 2024-08-06T17:40:04.506 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 23/34 2024-08-06T17:40:04.553 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 24/34 2024-08-06T17:40:04.631 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 25/34 2024-08-06T17:40:04.720 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 26/34 2024-08-06T17:40:04.761 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/34 2024-08-06T17:40:04.875 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 28/34 2024-08-06T17:40:05.031 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 29/34 2024-08-06T17:40:05.063 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:05.103 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 30/34 2024-08-06T17:40:05.103 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:05.103 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-08-06T17:40:05.103 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:05.104 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-08-06T17:40:05.104 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:05.289 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:05.315 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:05.315 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:05.315 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-06T17:40:05.315 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:05.316 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:05.316 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:05.689 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:05.713 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:05.713 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:05.713 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-06T17:40:05.713 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:05.714 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:05.714 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:05.851 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 1/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 3/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 4/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 5/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 6/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-06T17:40:05.852 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-06T17:40:05.853 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-06T17:40:05.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-06T17:40:05.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-06T17:40:05.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-06T17:40:05.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-06T17:40:05.854 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-06T17:40:05.855 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-06T17:40:05.855 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-06T17:40:05.855 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-06T17:40:05.932 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:05.955 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:05.956 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:05.956 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-06T17:40:05.956 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:05.956 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:05.956 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:06.020 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:06.465 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:06.494 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 31/34 2024-08-06T17:40:06.494 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:06.494 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-08-06T17:40:06.494 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:06.494 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-08-06T17:40:06.495 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:06.882 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:06.895 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-06T17:40:06.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:06.895 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-06T17:40:06.896 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-06T17:40:06.897 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:06.898 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:06.906 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 32/34 2024-08-06T17:40:06.906 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:06.906 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-08-06T17:40:06.906 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:06.907 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-08-06T17:40:06.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:07.108 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:07.131 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 33/34 2024-08-06T17:40:07.132 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:07.132 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-08-06T17:40:07.132 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:07.132 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-08-06T17:40:07.132 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:07.197 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:07.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 1/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 3/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 4/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 5/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 6/34 2024-08-06T17:40:07.646 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-06T17:40:07.647 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-06T17:40:07.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-06T17:40:07.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-06T17:40:07.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-06T17:40:07.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-06T17:40:07.648 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-06T17:40:07.649 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-06T17:40:07.650 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-06T17:40:07.823 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:49 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:07.909 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:07.951 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:07.952 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:07.952 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:08.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-08-06T17:40:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-06T17:40:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:08.456 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:08.456 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-06T17:40:08.457 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ply-3.11-14.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-06T17:40:08.458 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:08.459 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:08.573 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:50 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:08.702 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:08.703 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:08.703 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-08-06T17:40:08.703 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 82 k 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 372 k 2024-08-06T17:40:08.704 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:08.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 34/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:17.2.7-1236.gb2513712.el9.x86_64 1/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 3/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 4/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 5/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9. 6/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/34 2024-08-06T17:40:08.867 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 15/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 16/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 17/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 18/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 19/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 20/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 21/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 22/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 23/34 2024-08-06T17:40:08.868 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 24/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 25/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 26/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 27/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 28/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 29/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 30/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 31/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 32/34 2024-08-06T17:40:08.869 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 33/34 2024-08-06T17:40:08.917 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-17.2.7-1236.gb2513712.el9.noarch.rpm 386 kB/s | 82 kB 00:00 2024-08-06T17:40:08.918 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:08.918 INFO:teuthology.orchestra.run.smithi086.stdout:Total 383 kB/s | 82 kB 00:00 2024-08-06T17:40:08.918 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:08.923 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:08.923 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:08.925 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:08.926 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:08.993 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:09.385 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:50 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:09.440 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:09.471 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:09.512 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:09.513 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:09.513 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:09.545 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:09.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 34/34 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:09.769 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-08-06T17:40:09.770 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-08-06T17:40:09.771 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:09.772 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:09.888 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:10.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-base 2024-08-06T17:40:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:10.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:10.132 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:10.132 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:10.132 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:10.133 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:51 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:10.262 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 82 k 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:10.263 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 82 k 2024-08-06T17:40:10.264 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 372 k 2024-08-06T17:40:10.264 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:10.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-08-06T17:40:10.447 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-17.2.7-1236.gb2513712.el9.noarch.rpm 449 kB/s | 82 kB 00:00 2024-08-06T17:40:10.447 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:10.447 INFO:teuthology.orchestra.run.smithi107.stdout:Total 445 kB/s | 82 kB 00:00 2024-08-06T17:40:10.447 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:10.453 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:10.453 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:10.454 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:10.455 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:10.522 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:10.674 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:49 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:10.760 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-base-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:10.803 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:10.804 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:10.804 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:10.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install cephadm 2024-08-06T17:40:10.885 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:52 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:10.912 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:11.015 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1236.gb2513712.el9 ceph 137 k 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:11.016 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:11.017 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 137 k 2024-08-06T17:40:11.017 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 379 k 2024-08-06T17:40:11.017 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:11.017 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:11.239 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-17.2.7-1236.gb25137 618 kB/s | 137 kB 00:00 2024-08-06T17:40:11.240 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:11.240 INFO:teuthology.orchestra.run.smithi086.stdout:Total 613 kB/s | 137 kB 00:00 2024-08-06T17:40:11.241 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:11.248 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:11.248 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:11.304 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:11.304 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:11.353 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:11.418 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:50 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:11.458 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:11.532 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:11.545 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:17.2.7-1236.gb2513712.el9 ceph-noarch 82 k 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:11.546 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:11.547 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:11.547 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:11.547 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 82 k 2024-08-06T17:40:11.547 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 372 k 2024-08-06T17:40:11.547 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:11.557 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:11.557 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:11.557 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-06T17:40:11.557 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:11.592 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:11.742 INFO:teuthology.orchestra.run.smithi117.stdout:cephadm-17.2.7-1236.gb2513712.el9.noarch.rpm 428 kB/s | 82 kB 00:00 2024-08-06T17:40:11.742 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:11.743 INFO:teuthology.orchestra.run.smithi117.stdout:Total 425 kB/s | 82 kB 00:00 2024-08-06T17:40:11.743 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:11.745 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:11.745 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:11.747 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:11.748 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:11.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2024-08-06T17:40:11.820 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:12.230 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:12.325 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:12.360 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:53 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:12.427 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:12.491 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1236.gb2513712.el9 ceph 137 k 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:12.492 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:12.493 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 137 k 2024-08-06T17:40:12.493 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 379 k 2024-08-06T17:40:12.493 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:12.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-08-06T17:40:12.651 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:12.727 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-17.2.7-1236.gb25137 588 kB/s | 137 kB 00:00 2024-08-06T17:40:12.727 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:12.728 INFO:teuthology.orchestra.run.smithi107.stdout:Total 584 kB/s | 137 kB 00:00 2024-08-06T17:40:12.728 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:12.735 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:12.735 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:12.794 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:12.794 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:17.2.7-1236.gb2513712.el9.noarch 1/1 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:17.2.7-1236.gb2513712.el9.noarch 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:12.895 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:12.951 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:13.020 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:13.045 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:13.045 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:13.045 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-06T17:40:13.045 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:13.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-immutable-object-cache 2024-08-06T17:40:13.192 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:55 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:13.281 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:13.325 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:13.326 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:13.326 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:13.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-08-06T17:40:13.695 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:52 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:13.828 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:13.829 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:13.829 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-08-06T17:40:13.829 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:13.829 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:13.829 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1236.gb2513712.el9 ceph 137 k 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 137 k 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 379 k 2024-08-06T17:40:13.830 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:13.934 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:55 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:13.951 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:13.951 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:13.952 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:13.952 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:14.063 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 813 k 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:14.064 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 893 k 2024-08-06T17:40:14.065 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.6 M 2024-08-06T17:40:14.065 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:14.070 INFO:teuthology.orchestra.run.smithi117.stdout:ceph-immutable-object-cache-17.2.7-1236.gb25137 574 kB/s | 137 kB 00:00 2024-08-06T17:40:14.070 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:14.071 INFO:teuthology.orchestra.run.smithi117.stdout:Total 570 kB/s | 137 kB 00:00 2024-08-06T17:40:14.071 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:14.078 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:14.078 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:14.134 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:14.135 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:14.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2024-08-06T17:40:14.288 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:14.354 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:14.379 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:14.379 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:14.379 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-08-06T17:40:14.379 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:14.410 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): ceph-fuse-17.2.7-1236.gb2513712.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-08-06T17:40:14.618 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): fuse-2.9.9-16.el9.x86_64.rpm 162 kB/s | 80 kB 00:00 2024-08-06T17:40:14.619 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:14.619 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.6 MB/s | 893 kB 00:00 2024-08-06T17:40:14.631 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:14.653 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:14.653 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:14.735 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:14.735 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:14.755 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:55 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:14.841 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:14.884 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:14.885 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:14.886 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:14.931 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:14.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2024-08-06T17:40:15.054 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-06T17:40:15.115 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:15.294 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1236.gb2513712. 1/1 2024-08-06T17:40:15.294 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:15.294 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:15.295 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:15.295 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:15.295 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:15.512 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:56 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:15.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-mgr 2024-08-06T17:40:15.642 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:15.642 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:15.642 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2024-08-06T17:40:15.642 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:15.642 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 813 k 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout:Install 2 Packages 2024-08-06T17:40:15.643 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:15.644 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 893 k 2024-08-06T17:40:15.644 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 2.6 M 2024-08-06T17:40:15.644 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:15.780 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:15.781 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/2 2024-08-06T17:40:15.863 INFO:teuthology.orchestra.run.smithi107.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 764 kB/s | 80 kB 00:00 2024-08-06T17:40:16.034 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-06T17:40:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:16.035 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-06T17:40:16.035 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:16.035 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:16.038 INFO:teuthology.orchestra.run.smithi107.stdout:(2/2): ceph-fuse-17.2.7-1236.gb2513712.el9.x86_ 2.8 MB/s | 813 kB 00:00 2024-08-06T17:40:16.039 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:16.039 INFO:teuthology.orchestra.run.smithi107.stdout:Total 2.2 MB/s | 893 kB 00:00 2024-08-06T17:40:16.051 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:16.073 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:16.073 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:16.077 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:55 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:16.154 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:16.155 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:16.166 INFO:teuthology.orchestra.run.smithi117.stdout:Package ceph-mgr-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:16.210 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:16.211 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:16.211 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:16.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-08-06T17:40:16.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-fuse 2024-08-06T17:40:16.343 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:16.468 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-06T17:40:16.528 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:16.816 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:00:58 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:16.828 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:55 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:16.948 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 126 k 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 126 k 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 448 k 2024-08-06T17:40:16.949 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:16.957 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:16.958 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:16.958 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-08-06T17:40:16.958 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 813 k 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout:Install 2 Packages 2024-08-06T17:40:16.959 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:16.960 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 893 k 2024-08-06T17:40:16.960 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 2.6 M 2024-08-06T17:40:16.960 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:17.170 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-17.2.7-1236.gb2513712.el9.x86_64 574 kB/s | 126 kB 00:00 2024-08-06T17:40:17.170 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:17.171 INFO:teuthology.orchestra.run.smithi086.stdout:Total 570 kB/s | 126 kB 00:00 2024-08-06T17:40:17.171 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:17.176 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:17.177 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:17.188 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/2 2024-08-06T17:40:17.232 INFO:teuthology.orchestra.run.smithi117.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 499 kB/s | 80 kB 00:00 2024-08-06T17:40:17.232 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:17.232 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:17.365 INFO:teuthology.orchestra.run.smithi117.stdout:(2/2): ceph-fuse-17.2.7-1236.gb2513712.el9.x86_ 2.7 MB/s | 813 kB 00:00 2024-08-06T17:40:17.366 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:17.366 INFO:teuthology.orchestra.run.smithi117.stdout:Total 2.1 MB/s | 893 kB 00:00 2024-08-06T17:40:17.378 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:17.385 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:17.400 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:17.400 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:17.453 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:17.482 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:17.482 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:17.523 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:17.662 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:17.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2024-08-06T17:40:17.802 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-08-06T17:40:17.862 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:17.922 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:18.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:18.142 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:18.142 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:18.143 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:18.143 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:18.143 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:18.245 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:59 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:18.378 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:18.379 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:18.379 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 126 k 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 126 k 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 448 k 2024-08-06T17:40:18.380 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:18.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-08-06T17:40:18.541 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2/2 2024-08-06T17:40:18.541 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/2 2024-08-06T17:40:18.611 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-17.2.7-1236.gb2513712.el9.x86_64 545 kB/s | 126 kB 00:00 2024-08-06T17:40:18.611 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:18.612 INFO:teuthology.orchestra.run.smithi107.stdout:Total 542 kB/s | 126 kB 00:00 2024-08-06T17:40:18.612 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:18.617 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:18.617 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:18.672 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:18.673 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:18.812 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:18.813 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:18.897 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:18.936 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:00 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:19.023 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:19.066 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:19.067 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:19.067 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:19.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados-devel 2024-08-06T17:40:19.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-08-06T17:40:19.295 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:19.557 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:19.557 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:19.558 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:19.558 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:19.558 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:19.558 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:19.630 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:00:58 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:19.676 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:01 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:19.765 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 126 k 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:19.766 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:19.767 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 126 k 2024-08-06T17:40:19.767 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 448 k 2024-08-06T17:40:19.767 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:19.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2024-08-06T17:40:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:19.806 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-08-06T17:40:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:19.806 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:19.807 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 25 k 2024-08-06T17:40:19.808 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 110 k 2024-08-06T17:40:19.808 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:19.990 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-17.2.7-1236.gb2513712.el9.x86_6 137 kB/s | 25 kB 00:00 2024-08-06T17:40:19.990 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:19.990 INFO:teuthology.orchestra.run.smithi086.stdout:Total 136 kB/s | 25 kB 00:00 2024-08-06T17:40:19.990 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:19.992 INFO:teuthology.orchestra.run.smithi117.stdout:librados-devel-17.2.7-1236.gb2513712.el9.x86_64 560 kB/s | 126 kB 00:00 2024-08-06T17:40:19.993 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:19.993 INFO:teuthology.orchestra.run.smithi117.stdout:Total 556 kB/s | 126 kB 00:00 2024-08-06T17:40:19.994 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:19.994 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:19.994 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:19.995 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:19.995 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:19.999 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:19.999 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:20.056 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:20.057 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:20.068 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:20.184 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.212 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:20.296 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.352 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:01 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:20.440 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:20.483 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:20.484 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:20.484 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:20.527 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2024-08-06T17:40:20.679 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.745 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:20.746 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:20.746 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:20.746 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:20.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-08-06T17:40:20.921 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:20.921 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:20.921 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:20.921 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:20.922 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:20.922 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:21.099 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:02 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:21.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs2 2024-08-06T17:40:21.226 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:21.227 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:21.227 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-08-06T17:40:21.227 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 25 k 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 110 k 2024-08-06T17:40:21.228 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:21.390 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-17.2.7-1236.gb2513712.el9.x86_6 154 kB/s | 25 kB 00:00 2024-08-06T17:40:21.391 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:21.391 INFO:teuthology.orchestra.run.smithi107.stdout:Total 153 kB/s | 25 kB 00:00 2024-08-06T17:40:21.391 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:21.394 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:21.395 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:21.396 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:21.396 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:21.457 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:03 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:21.470 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:21.543 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:21.577 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:21.587 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:21.587 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:21.587 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:21.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-08-06T17:40:21.691 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:00 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:21.779 INFO:teuthology.orchestra.run.smithi117.stdout:Package libcephfs2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:21.823 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:21.824 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:21.824 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:21.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install libcephfs-devel 2024-08-06T17:40:21.916 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:22.139 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:22.194 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:04 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:22.278 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:22.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2024-08-06T17:40:22.320 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:22.321 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:22.321 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:22.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-08-06T17:40:22.427 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:01 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:22.555 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:17.2.7-1236.gb2513712.el9 ceph 25 k 2024-08-06T17:40:22.556 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 25 k 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 110 k 2024-08-06T17:40:22.557 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:22.721 INFO:teuthology.orchestra.run.smithi117.stdout:libcephfs-devel-17.2.7-1236.gb2513712.el9.x86_6 152 kB/s | 25 kB 00:00 2024-08-06T17:40:22.722 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:22.722 INFO:teuthology.orchestra.run.smithi117.stdout:Total 150 kB/s | 25 kB 00:00 2024-08-06T17:40:22.722 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:22.726 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:22.726 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:22.727 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:22.727 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:22.801 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:22.857 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:03 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:22.917 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:22.928 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:04 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:22.942 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:22.984 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:22.985 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:22.985 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:23.014 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:23.056 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:23.057 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:23.057 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:23.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2024-08-06T17:40:23.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-08-06T17:40:23.262 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:23.485 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:23.485 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:23.486 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:23.602 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:04 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:23.671 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:05 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:23.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librados2 2024-08-06T17:40:23.687 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:23.729 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:23.730 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:23.730 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:23.755 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:23.797 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:23.799 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:23.799 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:23.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2024-08-06T17:40:23.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-08-06T17:40:24.217 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:03 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:24.302 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:24.344 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:24.345 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:24.345 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:24.355 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:05 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:24.404 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:24.416 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install librbd1 2024-08-06T17:40:24.447 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:24.490 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:24.492 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:24.493 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:24.493 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:24.533 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:24.533 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:24.533 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:24.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2024-08-06T17:40:24.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-08-06T17:40:24.959 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:03 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:25.045 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:25.087 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:25.089 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:25.089 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:25.135 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:07 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:25.139 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:25.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rados 2024-08-06T17:40:25.221 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:25.229 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:25.263 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:25.264 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-08-06T17:40:25.264 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:25.273 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:25.274 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:25.274 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:25.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-08-06T17:40:25.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2024-08-06T17:40:25.694 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:04 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:25.780 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rados-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:25.822 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:25.823 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:25.823 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:25.875 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:07 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:25.897 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:25.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rgw 2024-08-06T17:40:25.983 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:26.003 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:26.003 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:26.003 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:26.003 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:26.003 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 82 k 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:26.004 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:26.005 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 82 k 2024-08-06T17:40:26.005 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 222 k 2024-08-06T17:40:26.005 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:26.025 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:26.026 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:26.026 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:26.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2024-08-06T17:40:26.210 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-17.2.7-1236.gb2513712.el9.x86_64.rpm 401 kB/s | 82 kB 00:00 2024-08-06T17:40:26.210 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:26.210 INFO:teuthology.orchestra.run.smithi086.stdout:Total 398 kB/s | 82 kB 00:00 2024-08-06T17:40:26.211 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:26.217 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:26.217 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:26.272 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:26.272 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:26.437 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:05 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:26.479 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:26.522 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rgw-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:26.563 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:26.565 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:26.566 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:26.567 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:26.641 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-cephfs 2024-08-06T17:40:26.641 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:07 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:26.726 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:26.768 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:26.769 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2024-08-06T17:40:26.769 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:26.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2024-08-06T17:40:26.894 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:27.181 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:27.182 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:27.266 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-cephfs-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:27.309 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:27.310 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:27.310 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:27.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-08-06T17:40:27.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install python3-rbd 2024-08-06T17:40:27.393 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:08 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:27.522 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:27.523 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:27.523 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:27.523 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:27.523 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 82 k 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 82 k 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 222 k 2024-08-06T17:40:27.524 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:27.727 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-17.2.7-1236.gb2513712.el9.x86_64.rpm 404 kB/s | 82 kB 00:00 2024-08-06T17:40:27.728 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:27.728 INFO:teuthology.orchestra.run.smithi107.stdout:Total 401 kB/s | 82 kB 00:00 2024-08-06T17:40:27.728 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:27.735 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:27.735 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:27.794 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:27.795 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:27.934 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:06 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:27.949 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:09 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:28.012 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:28.030 INFO:teuthology.orchestra.run.smithi117.stdout:Package python3-rbd-2:17.2.7-1236.gb2513712.el9.x86_64 is already installed. 2024-08-06T17:40:28.073 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:28.074 INFO:teuthology.orchestra.run.smithi117.stdout:Nothing to do. 2024-08-06T17:40:28.074 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:28.080 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:28.081 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:28.082 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:28.082 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:28.082 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.9 M 2024-08-06T17:40:28.082 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 12 M 2024-08-06T17:40:28.082 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:28.106 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:28.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-fuse 2024-08-06T17:40:28.430 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-17.2.7-1236.gb2513712.el9.x86_64.rpm 8.5 MB/s | 2.9 MB 00:00 2024-08-06T17:40:28.430 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:28.431 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.4 MB/s | 2.9 MB 00:00 2024-08-06T17:40:28.431 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:28.440 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:28.440 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:28.465 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:28.512 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:28.513 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:28.688 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:07 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:28.722 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:28.816 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:28.816 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:28.816 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:17.2.7-1236.gb2513712.el9 ceph 82 k 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:28.817 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 82 k 2024-08-06T17:40:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 222 k 2024-08-06T17:40:28.818 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:28.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2024-08-06T17:40:28.923 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:28.989 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:29.015 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:29.015 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:29.015 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-06T17:40:29.015 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:29.015 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:29.016 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:29.024 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-fuse-17.2.7-1236.gb2513712.el9.x86_64.rpm 397 kB/s | 82 kB 00:00 2024-08-06T17:40:29.025 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:29.025 INFO:teuthology.orchestra.run.smithi117.stdout:Total 394 kB/s | 82 kB 00:00 2024-08-06T17:40:29.025 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:29.031 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:29.031 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:29.087 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:29.087 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:29.282 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:29.461 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:29.472 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:10 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:29.602 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:29.602 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:29.603 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:29.604 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.9 M 2024-08-06T17:40:29.604 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 12 M 2024-08-06T17:40:29.604 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:29.732 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:29.957 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-17.2.7-1236.gb2513712.el9.x86_64.rpm 8.3 MB/s | 2.9 MB 00:00 2024-08-06T17:40:29.958 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:29.958 INFO:teuthology.orchestra.run.smithi107.stdout:Total 8.3 MB/s | 2.9 MB 00:00 2024-08-06T17:40:29.958 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:29.967 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:29.967 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:30.001 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:30.001 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:30.002 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:30.002 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:30.002 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:30.002 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:30.039 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:30.040 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:30.041 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:30.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-mirror 2024-08-06T17:40:30.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-08-06T17:40:30.457 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:30.508 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:30.533 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:30.779 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:09 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:30.801 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:12 ago on Tue 06 Aug 2024 05:39:18 PM UTC. 2024-08-06T17:40:30.908 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:30.909 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repo Size 2024-08-06T17:40:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:17.2.7-1236.gb2513712.el9 ceph 2.9 M 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 2.9 M 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 12 M 2024-08-06T17:40:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:30.933 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 165 k 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-08-06T17:40:30.934 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-08-06T17:40:30.935 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-08-06T17:40:30.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:30.935 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 165 k 2024-08-06T17:40:30.935 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 477 k 2024-08-06T17:40:30.935 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-08-06T17:40:31.166 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-17.2.7-1236.gb2513712.el9.x86_64.rpm 715 kB/s | 165 kB 00:00 2024-08-06T17:40:31.166 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:31.167 INFO:teuthology.orchestra.run.smithi086.stdout:Total 711 kB/s | 165 kB 00:00 2024-08-06T17:40:31.167 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-08-06T17:40:31.174 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-08-06T17:40:31.174 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-08-06T17:40:31.231 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-08-06T17:40:31.231 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-08-06T17:40:31.252 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-mirror-17.2.7-1236.gb2513712.el9.x86_64.rpm 8.6 MB/s | 2.9 MB 00:00 2024-08-06T17:40:31.253 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:31.253 INFO:teuthology.orchestra.run.smithi117.stdout:Total 8.6 MB/s | 2.9 MB 00:00 2024-08-06T17:40:31.253 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:31.262 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:31.263 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:31.334 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:31.334 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:31.376 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-08-06T17:40:31.469 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:31.547 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:31.744 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:31.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2024-08-06T17:40:31.803 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:31.827 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:31.828 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-08-06T17:40:31.828 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-08-06T17:40:31.828 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:31.828 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-08-06T17:40:31.828 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:31.859 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:32.079 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:32.079 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:32.080 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-08-06T17:40:32.080 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:32.080 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:40:32.080 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-08-06T17:40:32.300 DEBUG:teuthology.parallel:result is None 2024-08-06T17:40:32.320 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:13 ago on Tue 06 Aug 2024 05:39:19 PM UTC. 2024-08-06T17:40:32.446 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 165 k 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2024-08-06T17:40:32.447 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:32.448 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 165 k 2024-08-06T17:40:32.448 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 477 k 2024-08-06T17:40:32.448 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2024-08-06T17:40:32.669 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-17.2.7-1236.gb2513712.el9.x86_64.rpm 743 kB/s | 165 kB 00:00 2024-08-06T17:40:32.670 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:32.670 INFO:teuthology.orchestra.run.smithi107.stdout:Total 738 kB/s | 165 kB 00:00 2024-08-06T17:40:32.670 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2024-08-06T17:40:32.678 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2024-08-06T17:40:32.678 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2024-08-06T17:40:32.734 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2024-08-06T17:40:32.734 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2024-08-06T17:40:32.874 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:32.875 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:32.875 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:32.875 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:32.875 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:32.875 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:32.886 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2024-08-06T17:40:32.981 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:33.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install rbd-nbd 2024-08-06T17:40:33.424 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:33.672 INFO:teuthology.orchestra.run.smithi117.stdout:Last metadata expiration check: 0:01:12 ago on Tue 06 Aug 2024 05:39:21 PM UTC. 2024-08-06T17:40:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:33.704 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2024-08-06T17:40:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:33.704 INFO:teuthology.orchestra.run.smithi107.stdout: 2024-08-06T17:40:33.705 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2024-08-06T17:40:33.800 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-08-06T17:40:33.800 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout: Package Architecture Version Repository Size 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:17.2.7-1236.gb2513712.el9 ceph 165 k 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:================================================================================ 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Install 1 Package 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 165 k 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Installed size: 477 k 2024-08-06T17:40:33.801 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-08-06T17:40:33.941 DEBUG:teuthology.parallel:result is None 2024-08-06T17:40:34.022 INFO:teuthology.orchestra.run.smithi117.stdout:rbd-nbd-17.2.7-1236.gb2513712.el9.x86_64.rpm 747 kB/s | 165 kB 00:00 2024-08-06T17:40:34.023 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:40:34.023 INFO:teuthology.orchestra.run.smithi117.stdout:Total 742 kB/s | 165 kB 00:00 2024-08-06T17:40:34.023 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-08-06T17:40:34.031 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-08-06T17:40:34.031 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-08-06T17:40:34.086 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-08-06T17:40:34.086 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-08-06T17:40:34.236 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-08-06T17:40:34.334 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:34.708 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 1/1 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:17.2.7-1236.gb2513712.el9.x86_64 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:40:34.976 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-08-06T17:40:35.193 DEBUG:teuthology.parallel:result is None 2024-08-06T17:40:35.193 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:40:35.193 INFO:teuthology.packaging:ref: None 2024-08-06T17:40:35.193 INFO:teuthology.packaging:tag: None 2024-08-06T17:40:35.193 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:40:35.194 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:35.194 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:40:35.448 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-06T17:40:35.474 INFO:teuthology.orchestra.run.smithi086.stdout:17.2.7-1236.gb2513712.el9 2024-08-06T17:40:35.475 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1236.gb2513712.el9 2024-08-06T17:40:35.475 INFO:teuthology.task.install:The correct ceph version 17.2.7-1236.gb2513712 is installed. 2024-08-06T17:40:35.477 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:40:35.477 INFO:teuthology.packaging:ref: None 2024-08-06T17:40:35.477 INFO:teuthology.packaging:tag: None 2024-08-06T17:40:35.477 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:40:35.477 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:35.477 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:40:35.735 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-06T17:40:35.759 INFO:teuthology.orchestra.run.smithi107.stdout:17.2.7-1236.gb2513712.el9 2024-08-06T17:40:35.760 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1236.gb2513712.el9 2024-08-06T17:40:35.760 INFO:teuthology.task.install:The correct ceph version 17.2.7-1236.gb2513712 is installed. 2024-08-06T17:40:35.762 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-08-06T17:40:35.762 INFO:teuthology.packaging:ref: None 2024-08-06T17:40:35.762 INFO:teuthology.packaging:tag: None 2024-08-06T17:40:35.762 INFO:teuthology.packaging:branch: quincy 2024-08-06T17:40:35.762 INFO:teuthology.packaging:sha1: 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:35.762 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-08-06T17:40:36.007 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-08-06T17:40:36.029 INFO:teuthology.orchestra.run.smithi117.stdout:17.2.7-1236.gb2513712.el9 2024-08-06T17:40:36.030 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1236.gb2513712.el9 2024-08-06T17:40:36.030 INFO:teuthology.task.install:The correct ceph version 17.2.7-1236.gb2513712 is installed. 2024-08-06T17:40:36.031 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-08-06T17:40:36.032 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:40:36.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-06T17:40:36.062 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:40:36.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-06T17:40:36.092 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:40:36.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-08-06T17:40:36.119 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-08-06T17:40:36.119 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:40:36.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-08-06T17:40:36.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-06T17:40:36.215 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:40:36.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2024-08-06T17:40:36.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-06T17:40:36.311 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:40:36.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-08-06T17:40:36.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-08-06T17:40:36.401 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-08-06T17:40:36.401 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:40:36.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-06T17:40:36.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-06T17:40:36.495 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:40:36.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-06T17:40:36.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-06T17:40:36.595 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:40:36.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-08-06T17:40:36.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-08-06T17:40:36.684 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-08-06T17:40:36.684 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:40:36.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-08-06T17:40:36.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-06T17:40:36.778 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:40:36.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/stdin-killer 2024-08-06T17:40:36.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-06T17:40:36.874 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:40:36.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-08-06T17:40:36.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-08-06T17:40:36.963 INFO:teuthology.run_tasks:Running task cephadm... 2024-08-06T17:40:37.056 INFO:tasks.cephadm:Config: {'conf': {'mon': {'auth allow insecure global id reclaim': True, 'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1'} 2024-08-06T17:40:37.057 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:37.057 INFO:tasks.cephadm:Cluster fsid is fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:40:37.058 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-08-06T17:40:37.058 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.86', 'mon.b': '172.21.15.107', 'mon.c': '172.21.15.117'} 2024-08-06T17:40:37.058 INFO:tasks.cephadm:First mon is mon.a on smithi086 2024-08-06T17:40:37.058 INFO:tasks.cephadm:First mgr is y 2024-08-06T17:40:37.058 INFO:tasks.cephadm:Normalizing hostnames... 2024-08-06T17:40:37.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo hostname $(hostname -s) 2024-08-06T17:40:37.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo hostname $(hostname -s) 2024-08-06T17:40:37.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-08-06T17:40:37.141 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-08-06T17:40:37.141 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:37.296 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55759/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-5484-g93ef537d', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-5484.g93ef537d'}, 'url': 'https://3.chacra.ceph.com/r/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-08-04 22:42:05.821838', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'main', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-08-06T17:40:37.444 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref main, sha1 93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:40:37.445 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/x86_64/flavors/default/cephadm 2024-08-06T17:40:37.445 INFO:tasks.cephadm:Downloading cephadm from url: https://3.chacra.ceph.com/binaries/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/x86_64/flavors/default/cephadm 2024-08-06T17:40:37.446 DEBUG:teuthology.orchestra.run.smithi086:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:37.843 INFO:teuthology.orchestra.run.smithi086.stdout:-rw-r--r--. 1 ubuntu ubuntu 791381 Aug 6 17:40 /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:37.843 DEBUG:teuthology.orchestra.run.smithi107:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.271 INFO:teuthology.orchestra.run.smithi107.stdout:-rw-r--r--. 1 ubuntu ubuntu 791381 Aug 6 17:40 /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.271 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/main/93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.661 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 791381 Aug 6 17:40 /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.662 DEBUG:teuthology.orchestra.run.smithi086:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.681 DEBUG:teuthology.orchestra.run.smithi107:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.700 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-08-06T17:40:38.733 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 on all hosts... 2024-08-06T17:40:38.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 pull 2024-08-06T17:40:38.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 pull 2024-08-06T17:40:38.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 pull 2024-08-06T17:40:39.567 INFO:teuthology.orchestra.run.smithi086.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1... 2024-08-06T17:40:39.608 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1... 2024-08-06T17:40:39.622 INFO:teuthology.orchestra.run.smithi107.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1... 2024-08-06T17:42:07.284 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:42:07.284 INFO:teuthology.orchestra.run.smithi086.stdout: "ceph_version": "ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev)", 2024-08-06T17:42:07.284 INFO:teuthology.orchestra.run.smithi086.stdout: "image_id": "c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c", 2024-08-06T17:42:07.285 INFO:teuthology.orchestra.run.smithi086.stdout: "repo_digests": [ 2024-08-06T17:42:07.285 INFO:teuthology.orchestra.run.smithi086.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60" 2024-08-06T17:42:07.285 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:42:07.285 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:42:07.405 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-08-06T17:42:07.405 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev)", 2024-08-06T17:42:07.406 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c", 2024-08-06T17:42:07.406 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-08-06T17:42:07.406 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60" 2024-08-06T17:42:07.406 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-08-06T17:42:07.406 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout: "ceph_version": "ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev)", 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout: "image_id": "c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c", 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout: "repo_digests": [ 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60" 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:42:07.480 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:42:07.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph 2024-08-06T17:42:07.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph 2024-08-06T17:42:07.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-08-06T17:42:07.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /etc/ceph 2024-08-06T17:42:07.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /etc/ceph 2024-08-06T17:42:07.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-08-06T17:42:07.666 INFO:tasks.cephadm:Writing seed config... 2024-08-06T17:42:07.667 INFO:tasks.cephadm: override: [mon] auth allow insecure global id reclaim = True 2024-08-06T17:42:07.667 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] mon scrub interval = 300 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [global] mon client directed command retry = 5 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [global] mon election default strategy = 1 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [global] ms inject socket failures = 2500 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [global] ms tcp read timeout = 5 2024-08-06T17:42:07.668 INFO:tasks.cephadm: override: [global] osd_pool_default_min_size = 2 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [global] osd_pool_default_size = 2 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] bluestore zero block detection = True 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd blocked scrub grace period = 3600 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd debug verify cached snaps = True 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd debug verify missing on start = True 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd map cache size = 1 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd max backfills = 6 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd max markdown count = 1000 2024-08-06T17:42:07.669 INFO:tasks.cephadm: override: [osd] osd mclock override recovery settings = True 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd mclock profile = high_recovery_ops 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd op queue = debug_random 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd op queue cut off = debug_random 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd scrub during recovery = False 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-08-06T17:42:07.670 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-08-06T17:42:07.671 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:42:07.671 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-08-06T17:42:07.687 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = fd7e697e-541a-11ef-bccc-c7b262605968 mon client directed command retry = 5 mon election default strategy = 1 ms inject socket failures = 2500 ms tcp read timeout = 5 osd_pool_default_min_size = 2 osd_pool_default_size = 2 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = True osd debug verify missing on start = True osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore zero block detection = True debug ms = 1 debug osd = 20 osd blocked scrub grace period = 3600 osd map cache size = 1 osd max backfills = 6 osd max markdown count = 1000 osd mclock override recovery settings = True osd scrub during recovery = False osd scrub min interval = 60 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false auth allow insecure global id reclaim = True debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 mon scrub interval = 300 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-08-06T17:42:07.687 DEBUG:teuthology.orchestra.run.smithi086:mon.a> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service 2024-08-06T17:42:07.729 DEBUG:teuthology.orchestra.run.smithi086:mgr.y> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.y.service 2024-08-06T17:42:07.772 INFO:tasks.cephadm:Bootstrapping... 2024-08-06T17:42:07.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 -v bootstrap --fsid fd7e697e-541a-11ef-bccc-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.86 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-08-06T17:42:07.976 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-08-06T17:42:07.976 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1', '-v', 'bootstrap', '--fsid', 'fd7e697e-541a-11ef-bccc-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.86', '--skip-admin-label'] 2024-08-06T17:42:07.976 INFO:teuthology.orchestra.run.smithi086.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-08-06T17:42:07.977 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying podman|docker is present... 2024-08-06T17:42:08.001 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.1.2 2024-08-06T17:42:08.001 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying lvm2 is present... 2024-08-06T17:42:08.001 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying time synchronization is in place... 2024-08-06T17:42:08.009 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-06T17:42:08.009 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-06T17:42:08.015 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-06T17:42:08.015 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:42:08.022 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-08-06T17:42:08.028 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-08-06T17:42:08.028 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-08-06T17:42:08.028 INFO:teuthology.orchestra.run.smithi086.stdout:Repeating the final host check... 2024-08-06T17:42:08.051 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout 5.1.2 2024-08-06T17:42:08.052 INFO:teuthology.orchestra.run.smithi086.stdout:podman (/bin/podman) version 5.1.2 is present 2024-08-06T17:42:08.052 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl is present 2024-08-06T17:42:08.052 INFO:teuthology.orchestra.run.smithi086.stdout:lvcreate is present 2024-08-06T17:42:08.058 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-08-06T17:42:08.058 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-08-06T17:42:08.064 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-08-06T17:42:08.064 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:42:08.071 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout enabled 2024-08-06T17:42:08.077 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout active 2024-08-06T17:42:08.077 INFO:teuthology.orchestra.run.smithi086.stdout:Unit chronyd.service is enabled and running 2024-08-06T17:42:08.077 INFO:teuthology.orchestra.run.smithi086.stdout:Host looks OK 2024-08-06T17:42:08.077 INFO:teuthology.orchestra.run.smithi086.stdout:Cluster fsid: fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Acquiring lock 140173174037760 on /run/cephadm/fd7e697e-541a-11ef-bccc-c7b262605968.lock 2024-08-06T17:42:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Lock 140173174037760 acquired on /run/cephadm/fd7e697e-541a-11ef-bccc-c7b262605968.lock 2024-08-06T17:42:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 3300 ... 2024-08-06T17:42:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying IP 172.21.15.86 port 6789 ... 2024-08-06T17:42:08.078 INFO:teuthology.orchestra.run.smithi086.stdout:Base mon IP(s) is [172.21.15.86:3300, 172.21.15.86:6789], mon addrv is [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-08-06T17:42:08.081 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.86 metric 100 2024-08-06T17:42:08.081 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.86 metric 100 2024-08-06T17:42:08.084 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-08-06T17:42:08.084 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-08-06T17:42:08.084 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1782sec hoplimit 64 pref medium 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d809/64 scope link 2024-08-06T17:42:08.087 INFO:teuthology.orchestra.run.smithi086.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-08-06T17:42:08.088 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-08-06T17:42:08.088 INFO:teuthology.orchestra.run.smithi086.stdout:Mon IP `172.21.15.86` is in CIDR network `172.21.0.0/20` 2024-08-06T17:42:08.088 INFO:teuthology.orchestra.run.smithi086.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-08-06T17:42:08.088 INFO:teuthology.orchestra.run.smithi086.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-08-06T17:42:08.089 INFO:teuthology.orchestra.run.smithi086.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1... 2024-08-06T17:42:08.585 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stdout c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c 2024-08-06T17:42:08.585 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1... 2024-08-06T17:42:08.585 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Getting image source signatures 2024-08-06T17:42:08.586 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:259ad5367ecf2990ac75e929d9c35164e91507fcc2f14b0e3b176a58394054b3 2024-08-06T17:42:08.586 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-08-06T17:42:08.586 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Copying config sha256:c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c 2024-08-06T17:42:08.586 INFO:teuthology.orchestra.run.smithi086.stdout:/bin/podman: stderr Writing manifest to image destination 2024-08-06T17:42:09.456 INFO:teuthology.orchestra.run.smithi086.stdout:ceph: stdout ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev) 2024-08-06T17:42:09.457 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph version: ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev) 2024-08-06T17:42:09.457 INFO:teuthology.orchestra.run.smithi086.stdout:Extracting ceph user uid/gid from container image... 2024-08-06T17:42:10.254 INFO:teuthology.orchestra.run.smithi086.stdout:stat: stdout 167 167 2024-08-06T17:42:10.254 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial keys... 2024-08-06T17:42:11.292 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQByYLJmnf04LhAA8b9Hd0FDFZfGaMt4Te/18A== 2024-08-06T17:42:12.222 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQBzYLJmHcfoIxAAf9lqrjoarR3qjWjl/wYNPA== 2024-08-06T17:42:13.112 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph-authtool: stdout AQB0YLJmo0G9HRAAOxcjTeOkhfHSaii2wD8Zyw== 2024-08-06T17:42:13.112 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial monmap... 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool for a [v2:172.21.15.86:3300,v1:172.21.15.86:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = quincy 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: set fsid to fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:14.051 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-08-06T17:42:14.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:42:14.052 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mon... 2024-08-06T17:42:14.973 INFO:teuthology.orchestra.run.smithi086.stdout:create mon.a on 2024-08-06T17:42:15.175 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-08-06T17:42:15.325 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-08-06T17:42:15.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target → /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target. 2024-08-06T17:42:15.501 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target → /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target. 2024-08-06T17:42:15.734 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a 2024-08-06T17:42:15.734 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service: Unit ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service not loaded. 2024-08-06T17:42:15.901 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target.wants/ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service → /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968@.service. 2024-08-06T17:42:16.396 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-06T17:42:16.397 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon to start... 2024-08-06T17:42:16.404 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mon... 2024-08-06T17:42:16.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:16 smithi086 systemd[1]: Started Ceph mon.a for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:42:17.451 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout cluster: 2024-08-06T17:42:17.451 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout id: fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:17.451 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-08-06T17:42:17.451 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout services: 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.541125s) 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout data: 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout pgs: 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:mon is available 2024-08-06T17:42:17.452 INFO:teuthology.orchestra.run.smithi086.stdout:Assimilating anything we can from ceph.conf... 2024-08-06T17:42:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-06T17:42:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: monmap epoch 1 2024-08-06T17:42:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: last_changed 2024-08-06T17:42:13.412697+0000 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: min_mon_release 19 (squid) 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: election_strategy: 1 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: fsmap 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: osdmap e1: 0 total, 0 up, 0 in 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: mgrmap e1: no daemons active 2024-08-06T17:42:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:17 smithi086 ceph-mon[28859]: from='client.? 172.21.15.86:0/4271080990' entity='client.admin' cmd={"prefix": "status"} : dispatch 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-06T17:42:18.482 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-06T17:42:18.483 INFO:teuthology.orchestra.run.smithi086.stdout:Generating new minimal ceph.conf... 2024-08-06T17:42:18.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:18 smithi086 ceph-mon[28859]: from='client.? 172.21.15.86:0/2964725860' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-06T17:42:18.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:18 smithi086 ceph-mon[28859]: from='client.? 172.21.15.86:0/2964725860' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-08-06T17:42:19.445 INFO:teuthology.orchestra.run.smithi086.stdout:Restarting the monitor... 2024-08-06T17:42:19.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:19 smithi086 ceph-mon[28859]: from='client.? 172.21.15.86:0/3132959279' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:42:19.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:19 smithi086 systemd[1]: Stopping Ceph mon.a for fd7e697e-541a-11ef-bccc-c7b262605968... 2024-08-06T17:42:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:19 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a[28855]: 2024-08-06T17:42:19.712+0000 7f68f5611640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-08-06T17:42:20.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:19 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a[28855]: 2024-08-06T17:42:19.712+0000 7f68f5611640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-08-06T17:42:20.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:19 smithi086 podman[29100]: 2024-08-06 17:42:19.877255783 +0000 UTC m=+0.266269505 container died 0c9065cc7d8d777b73aa29bf8dc696f486a18317ec4c9f7d983485ca38d4200b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, RELEASE=main-93ef537, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-06T17:42:20.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 podman[29100]: 2024-08-06 17:42:20.204737124 +0000 UTC m=+0.593750845 container cleanup 0c9065cc7d8d777b73aa29bf8dc696f486a18317ec4c9f7d983485ca38d4200b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=main-93ef537, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-08-06T17:42:20.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 bash[29100]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a 2024-08-06T17:42:20.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 podman[29112]: 2024-08-06 17:42:20.397359146 +0000 UTC m=+0.512921777 container remove 0c9065cc7d8d777b73aa29bf8dc696f486a18317ec4c9f7d983485ca38d4200b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-93ef537, org.label-schema.vendor=CentOS) 2024-08-06T17:42:20.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service: Deactivated successfully. 2024-08-06T17:42:20.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 systemd[1]: Stopped Ceph mon.a for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:42:20.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 systemd[1]: Starting Ceph mon.a for fd7e697e-541a-11ef-bccc-c7b262605968... 2024-08-06T17:42:21.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 podman[29210]: 2024-08-06 17:42:20.823024468 +0000 UTC m=+0.109542831 container create 5fb100e791e68d1da1b72d6c4a3556b9d060959d060b92b62ab4700e463d7a5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-06T17:42:21.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:20 smithi086 podman[29210]: 2024-08-06 17:42:20.730288993 +0000 UTC m=+0.016807364 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 2024-08-06T17:42:21.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 podman[29210]: 2024-08-06 17:42:21.025364841 +0000 UTC m=+0.311883205 container init 5fb100e791e68d1da1b72d6c4a3556b9d060959d060b92b62ab4700e463d7a5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, org.label-schema.build-date=20240731, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 podman[29210]: 2024-08-06 17:42:21.02845755 +0000 UTC m=+0.314975917 container start 5fb100e791e68d1da1b72d6c4a3556b9d060959d060b92b62ab4700e463d7a5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , RELEASE=main-93ef537, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: set uid:gid to 167:167 (ceph:ceph) 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: ceph version 19.0.0-5484-g93ef537d (93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1) squid (dev), process ceph-mon, pid 2 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: pidfile_write: ignore empty --pid-file 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: load: jerasure load: lrc 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: RocksDB version: 7.9.2 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Git sha 0 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Compile date 2024-08-04 21:16:54 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: DB SUMMARY 2024-08-06T17:42:21.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: DB Session ID: B9WQNUIDME7KCWPPVWTU 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: CURRENT file: CURRENT 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: IDENTITY file: IDENTITY 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 97645 ; 2024-08-06T17:42:21.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.error_if_exists: 0 2024-08-06T17:42:21.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.create_if_missing: 0 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.paranoid_checks: 1 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.flush_verify_memtable_count: 1 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.env: 0x55c410e82a40 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.fs: PosixFileSystem 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.info_log: 0x55c413581520 2024-08-06T17:42:21.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_file_opening_threads: 16 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.statistics: (nil) 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.use_fsync: 0 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_log_file_size: 0 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.log_file_time_to_roll: 0 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.keep_log_file_num: 1000 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.recycle_log_file_num: 0 2024-08-06T17:42:21.095 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_fallocate: 1 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_mmap_reads: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_mmap_writes: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.use_direct_reads: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.create_missing_column_families: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.db_log_dir: 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.wal_dir: 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.table_cache_numshardbits: 6 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.WAL_ttl_seconds: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.WAL_size_limit_MB: 0 2024-08-06T17:42:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.is_fd_close_on_exec: 1 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.advise_random_on_open: 1 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.db_write_buffer_size: 0 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.write_buffer_manager: 0x55c413584e60 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.use_adaptive_mutex: 0 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.rate_limiter: (nil) 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.wal_recovery_mode: 2 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enable_thread_tracking: 0 2024-08-06T17:42:21.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enable_pipelined_write: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.unordered_write: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.row_cache: None 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.wal_filter: None 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_ingest_behind: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.two_write_queues: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.manual_wal_flush: 0 2024-08-06T17:42:21.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.wal_compression: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.atomic_flush: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.persist_stats_to_disk: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.write_dbid_to_manifest: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.log_readahead_size: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.best_efforts_recovery: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.allow_data_in_errors: 0 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.db_host_id: __hostname__ 2024-08-06T17:42:21.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enforce_single_del_contracts: true 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_background_jobs: 2 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_background_compactions: -1 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_subcompactions: 1 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.delayed_write_rate : 16777216 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_total_wal_size: 0 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.stats_dump_period_sec: 600 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.stats_persist_period_sec: 600 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_open_files: -1 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bytes_per_sync: 0 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.wal_bytes_per_sync: 0 2024-08-06T17:42:21.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.strict_bytes_per_sync: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_readahead_size: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_background_flushes: -1 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Compression algorithms supported: 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kZSTD supported: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kXpressCompression supported: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kBZip2Compression supported: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kLZ4Compression supported: 1 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kZlibCompression supported: 1 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kLZ4HCCompression supported: 1 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: kSnappyCompression supported: 1 2024-08-06T17:42:21.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Fast CRC32 supported: Supported on x86 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: DMutex implementation: pthread_mutex_t 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.merge_operator: 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_filter: None 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_filter_factory: None 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.sst_partitioner_factory: None 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.memtable_factory: SkipListFactory 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.table_factory: BlockBasedTable 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c413581120) 2024-08-06T17:42:21.102 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks: 1 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: pin_top_level_index_and_filter: 1 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_type: 0 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_index_type: 0 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_shortening: 1 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: data_block_hash_table_util_ratio: 0.750000 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: checksum: 4 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: no_block_cache: 0 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache: 0x55c413575610 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_name: BinnedLRUCache 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_options: 2024-08-06T17:42:21.103 INFO:journalctl@ceph.mon.a.smithi086.stdout: capacity : 536870912 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: num_shard_bits : 4 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: strict_capacity_limit : 0 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: high_pri_pool_ratio: 0.000 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_cache_compressed: (nil) 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: persistent_cache: (nil) 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size: 4096 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_size_deviation: 10 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_restart_interval: 16 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: index_block_restart_interval: 1 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: metadata_block_size: 4096 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: partition_filters: 0 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: use_delta_encoding: 1 2024-08-06T17:42:21.104 INFO:journalctl@ceph.mon.a.smithi086.stdout: filter_policy: bloomfilter 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: whole_key_filtering: 1 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: verify_compression: 0 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: read_amp_bytes_per_bit: 0 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: format_version: 5 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_index_compression: 1 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: block_align: 0 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: max_auto_readahead_size: 262144 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: prepopulate_block_cache: 0 2024-08-06T17:42:21.105 INFO:journalctl@ceph.mon.a.smithi086.stdout: initial_auto_readahead_size: 8192 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout: num_file_reads_for_auto_readahead: 2 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.write_buffer_size: 33554432 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_write_buffer_number: 2 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression: NoCompression 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression: Disabled 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.prefix_extractor: nullptr 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.num_levels: 7 2024-08-06T17:42:21.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-08-06T17:42:21.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.window_bits: -14 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.level: 32767 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.strategy: 0 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.enabled: false 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-08-06T17:42:21.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.target_file_size_base: 67108864 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.target_file_size_multiplier: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-08-06T17:42:21.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.arena_block_size: 1048576 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.disable_auto_compactions: 0 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-08-06T17:42:21.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.inplace_update_support: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.inplace_update_num_locks: 10000 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.memtable_huge_page_size: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.bloom_locality: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.max_successive_merges: 0 2024-08-06T17:42:21.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.optimize_filters_for_hits: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.paranoid_file_checks: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.force_consistency_checks: 1 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.report_bg_io_stats: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.ttl: 2592000 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.periodic_compaction_seconds: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enable_blob_files: false 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.min_blob_size: 0 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_file_size: 268435456 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_compression_type: NoCompression 2024-08-06T17:42:21.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.enable_blob_garbage_collection: false 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.blob_file_starting_level: 0 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: efac0c3f-0e4f-4cec-9503-39780d9763e3 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722966141054117, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722966141055112, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 93385, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 292, "table_properties": {"data_size": 91433, "index_size": 272, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 12408, "raw_average_key_size": 49, "raw_value_size": 84579, "raw_average_value_size": 336, "num_data_blocks": 12, "num_entries": 251, "num_filter_entries": 251, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1722966141, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "efac0c3f-0e4f-4cec-9503-39780d9763e3", "db_session_id": "B9WQNUIDME7KCWPPVWTU", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-08-06T17:42:21.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: EVENT_LOG_v1 {"time_micros": 1722966141055211, "job": 1, "event": "recovery_finished"} 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c4135a4e00 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: DB pointer 0x55c4136a2000 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** DB Stats ** 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-06T17:42:21.114 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: L0 2/0 93.06 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Sum 2/0 93.06 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 134.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** Compaction Stats [default] ** 2024-08-06T17:42:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 134.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: AddFile(Keys): cumulative 0, interval 0 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Cumulative compaction: 0.00 GB write, 28.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Interval compaction: 0.00 GB write, 28.18 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-08-06T17:42:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: Block cache BinnedLRUCache@0x55c413575610#2 capacity: 512.00 MB usage: 1.30 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: ** File Read Latency Histogram By Level [default] ** 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: starting mon.a rank 0 at public addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] at bind addrs [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???) e1 preinit fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).mds e1 new map 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).mds e1 print_map 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: e1 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: btime 2024-08-06T17:42:16:357156+0000 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: legacy client fscid: -1 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: 2024-08-06T17:42:21.117 INFO:journalctl@ceph.mon.a.smithi086.stdout: No filesystems configured 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 bash[29210]: 5fb100e791e68d1da1b72d6c4a3556b9d060959d060b92b62ab4700e463d7a5c 2024-08-06T17:42:21.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 systemd[1]: Started Ceph mon.a for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:42:21.118 INFO:teuthology.orchestra.run.smithi086.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mon.a is new leader, mons a in quorum (ranks 0) 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: monmap epoch 1 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: last_changed 2024-08-06T17:42:13.412697+0000 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: min_mon_release 19 (squid) 2024-08-06T17:42:21.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: election_strategy: 1 2024-08-06T17:42:21.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:42:21.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: fsmap 2024-08-06T17:42:21.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: osdmap e1: 0 total, 0 up, 0 in 2024-08-06T17:42:21.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:21 smithi086 ceph-mon[29229]: mgrmap e1: no daemons active 2024-08-06T17:42:22.132 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote config to /etc/ceph/ceph.conf 2024-08-06T17:42:22.154 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-08-06T17:42:22.154 INFO:teuthology.orchestra.run.smithi086.stdout:Creating mgr... 2024-08-06T17:42:22.155 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying port 0.0.0.0:9283 ... 2024-08-06T17:42:22.155 INFO:teuthology.orchestra.run.smithi086.stdout:Verifying port 0.0.0.0:8765 ... 2024-08-06T17:42:22.390 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.y 2024-08-06T17:42:22.390 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Failed to reset failed state of unit ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.y.service: Unit ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.y.service not loaded. 2024-08-06T17:42:22.528 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968.target.wants/ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.y.service → /etc/systemd/system/ceph-fd7e697e-541a-11ef-bccc-c7b262605968@.service. 2024-08-06T17:42:22.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1726880324' entity='client.admin' 2024-08-06T17:42:23.034 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:22 smithi086 podman[29441]: 2024-08-06 17:42:22.955216601 +0000 UTC m=+0.301702469 container start 54ca3613eac5aa299187f3f01fd148ca54fc465e493ccd9555955cd54d0e82e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=main-93ef537, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS) 2024-08-06T17:42:23.034 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:23 smithi086 bash[29441]: 54ca3613eac5aa299187f3f01fd148ca54fc465e493ccd9555955cd54d0e82e1 2024-08-06T17:42:23.047 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-06T17:42:23.047 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-08-06T17:42:23.055 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-06T17:42:23.055 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:42:23.055 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-08-06T17:42:23.055 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to enable service . firewalld.service is not available 2024-08-06T17:42:23.064 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-06T17:42:23.064 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr to start... 2024-08-06T17:42:23.072 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr... 2024-08-06T17:42:23.287 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:23 smithi086 systemd[1]: Started Ceph mgr.y for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:42:23.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1026426022' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "fd7e697e-541a-11ef-bccc-c7b262605968", 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.087 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "a" 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-06T17:42:24.088 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-06T17:42:24.089 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-06T17:42:24.092 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "btime": "2024-08-06T17:42:16:357156+0000", 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": false, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-06T17:42:24.093 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-08-06T17:42:16.357811+0000", 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-06T17:42:24.094 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:42:24.095 INFO:teuthology.orchestra.run.smithi086.stdout:mgr not available, waiting (1/15)... 2024-08-06T17:42:28.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: Activating manager daemon y 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: mgrmap e2: y(active, starting, since 0.0032757s) 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: Manager daemon y is now available 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' 2024-08-06T17:42:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:27 smithi086 ceph-mon[29229]: from='mgr.14100 172.21.15.86:0/3764550413' entity='mgr.y' 2024-08-06T17:42:30.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:29 smithi086 ceph-mon[29229]: mgrmap e3: y(active, since 1.00798s) 2024-08-06T17:42:30.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3642563227' entity='client.admin' cmd={"prefix": "status", "format": "json-pretty"} : dispatch 2024-08-06T17:42:31.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:30 smithi086 ceph-mon[29229]: mgrmap e4: y(active, since 2s) 2024-08-06T17:42:35.515 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:35.515 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:42:35.515 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsid": "fd7e697e-541a-11ef-bccc-c7b262605968", 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "health": { 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 0 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "a" 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-08-06T17:42:35.516 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "monmap": { 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.517 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "btime": "2024-08-06T17:42:16:357156+0000", 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-08-06T17:42:35.518 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modules": [ 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "iostat", 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "nfs", 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "restful" 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ], 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "modified": "2024-08-06T17:42:16.357811+0000", 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "services": {} 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout }, 2024-08-06T17:42:35.519 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-08-06T17:42:35.520 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:42:35.520 INFO:teuthology.orchestra.run.smithi086.stdout:mgr is available 2024-08-06T17:42:38.506 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1347615262' entity='client.admin' cmd={"prefix": "config assimilate-conf"} : dispatch 2024-08-06T17:42:38.766 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:38.768 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [global] 2024-08-06T17:42:38.768 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout fsid = fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.86:3300,v1:172.21.15.86:6789] 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ms_tcp_read_timeout = 5 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [mgr] 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout [osd] 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-08-06T17:42:38.769 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-08-06T17:42:38.770 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling cephadm module... 2024-08-06T17:42:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/284186462' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "cephadm"} : dispatch 2024-08-06T17:42:39.866 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:39 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ignoring --setuser ceph since I am not root 2024-08-06T17:42:39.867 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:39 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ignoring --setgroup ceph since I am not root 2024-08-06T17:42:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/284186462' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-08-06T17:42:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:40 smithi086 ceph-mon[29229]: mgrmap e5: y(active, since 11s) 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-08-06T17:42:41.272 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 5... 2024-08-06T17:42:41.597 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2709119817' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-06T17:42:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: Active manager daemon y restarted 2024-08-06T17:42:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: Activating manager daemon y 2024-08-06T17:42:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: osdmap e2: 0 total, 0 up, 0 in 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: mgrmap e6: y(active, starting, since 0.00308931s) 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-06T17:42:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:44 smithi086 ceph-mon[29229]: Manager daemon y is now available 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:45.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:45.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-06T17:42:45.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-06T17:42:45.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-06T17:42:45.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: mgrmap e7: y(active, since 1.00771s) 2024-08-06T17:42:45.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:45 smithi086 ceph-mon[29229]: from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-06T17:42:45.997 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:42:45.998 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-08-06T17:42:45.998 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-06T17:42:45.998 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:42:45.998 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 5 is available 2024-08-06T17:42:45.998 INFO:teuthology.orchestra.run.smithi086.stdout:Setting orchestrator backend to cephadm... 2024-08-06T17:42:46.594 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: 2024-08-06T17:42:46.542+0000 7f7c586a0640 -1 log_channel(cephadm) log [ERR] : [06/Aug/2024:17:42:46] ENGINE Error in HTTPServer.serve 2024-08-06T17:42:46.594 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: Traceback (most recent call last): 2024-08-06T17:42:46.594 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-06T17:42:46.594 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._connections.run(self.expiration_interval) 2024-08-06T17:42:46.594 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._run(expiration_interval) 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: new_conn = self._from_server_socket(self.server.socket) 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: s = self.context.wrap_socket( 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: return self.sslsocket_class._create( 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self.do_handshake() 2024-08-06T17:42:46.595 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-06T17:42:46.596 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._sslobj.do_handshake() 2024-08-06T17:42:46.596 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-06T17:42:46.596 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: 2024-08-06T17:42:47.269 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-mon[29229]: [06/Aug/2024:17:42:46] ENGINE Bus STARTING 2024-08-06T17:42:47.269 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:47.269 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:47.269 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:46 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: [06/Aug/2024:17:42:46] ENGINE Serving on https://172.21.15.86:7150 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: [06/Aug/2024:17:42:46] ENGINE Error in HTTPServer.serve 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-08-06T17:42:48.249 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-08-06T17:42:48.250 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-06T17:42:48.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:48.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: [06/Aug/2024:17:42:46] ENGINE Serving on http://172.21.15.86:8765 2024-08-06T17:42:48.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: [06/Aug/2024:17:42:46] ENGINE Bus STARTED 2024-08-06T17:42:48.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:47 smithi086 ceph-mon[29229]: mgrmap e8: y(active, since 2s) 2024-08-06T17:42:48.251 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout value unchanged 2024-08-06T17:42:48.252 INFO:teuthology.orchestra.run.smithi086.stdout:Generating ssh key... 2024-08-06T17:42:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:48 smithi086 ceph-mon[29229]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: Generating public/private rsa key pair. 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: Your identification has been saved in /tmp/tmp_nw2jfp8/key 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: Your public key has been saved in /tmp/tmp_nw2jfp8/key.pub 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: The key fingerprint is: 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: SHA256:Ja5m3PaRiCbt42DMYQ14tB2E3XHqEapW3ytGdvN7DCQ ceph-fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: The key's randomart image is: 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: +---[RSA 3072]----+ 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | .+o.o.. | 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | o.o.o.+ | 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | . + + + . | 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | . = + =E . | 2024-08-06T17:42:49.367 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | = . S +o | 2024-08-06T17:42:49.368 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | = + * o =. | 2024-08-06T17:42:49.368 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | * O * + .o | 2024-08-06T17:42:49.368 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | . B.o o . .o | 2024-08-06T17:42:49.368 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: | .o. . .. | 2024-08-06T17:42:49.368 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:42:49 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: +----[SHA256]-----+ 2024-08-06T17:42:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:50 smithi086 ceph-mon[29229]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:50 smithi086 ceph-mon[29229]: Generating ssh key... 2024-08-06T17:42:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:50 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:50 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:50.847 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:42:50.847 INFO:teuthology.orchestra.run.smithi086.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-08-06T17:42:50.847 INFO:teuthology.orchestra.run.smithi086.stdout:Adding key to root@localhost authorized_keys... 2024-08-06T17:42:50.848 INFO:teuthology.orchestra.run.smithi086.stdout:Adding host smithi086... 2024-08-06T17:42:51.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:51 smithi086 ceph-mon[29229]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:52.215 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:52 smithi086 ceph-mon[29229]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi086", "addr": "172.21.15.86", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:53.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:53 smithi086 ceph-mon[29229]: Deploying cephadm binary to smithi086 2024-08-06T17:42:55.215 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Added host 'smithi086' with addr '172.21.15.86' 2024-08-06T17:42:55.215 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying unmanaged mon service... 2024-08-06T17:42:55.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:55 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:55.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:55 smithi086 ceph-mon[29229]: Added host smithi086 2024-08-06T17:42:55.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:55 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:42:56.668 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-08-06T17:42:56.668 INFO:teuthology.orchestra.run.smithi086.stdout:Deploying unmanaged mgr service... 2024-08-06T17:42:57.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:56 smithi086 ceph-mon[29229]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:42:57.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:56 smithi086 ceph-mon[29229]: Saving service mon spec with placement count:5 2024-08-06T17:42:57.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:56 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:57.919 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-08-06T17:42:58.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:58 smithi086 ceph-mon[29229]: Saving service mgr spec with placement count:2 2024-08-06T17:42:58.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:58 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:58.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:58 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:42:58.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:58 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:59.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3656293495' entity='client.admin' 2024-08-06T17:42:59.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:59 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:42:59.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:42:59 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:43:01.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1415410611' entity='client.admin' 2024-08-06T17:43:01.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:00 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:43:01.095 INFO:teuthology.orchestra.run.smithi086.stdout:Enabling the dashboard module... 2024-08-06T17:43:02.853 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:02 smithi086 ceph-mon[29229]: from='mgr.14116 172.21.15.86:0/598037277' entity='mgr.y' 2024-08-06T17:43:02.853 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3265514574' entity='client.admin' cmd={"prefix": "mgr module enable", "module": "dashboard"} : dispatch 2024-08-06T17:43:02.853 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:02 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ignoring --setuser ceph since I am not root 2024-08-06T17:43:02.854 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:02 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ignoring --setgroup ceph since I am not root 2024-08-06T17:43:03.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3265514574' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-08-06T17:43:03.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:03 smithi086 ceph-mon[29229]: mgrmap e9: y(active, since 18s) 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "available": true, 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for the mgr to restart... 2024-08-06T17:43:04.266 INFO:teuthology.orchestra.run.smithi086.stdout:Waiting for mgr epoch 9... 2024-08-06T17:43:04.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2233951160' entity='client.admin' cmd={"prefix": "mgr stat"} : dispatch 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: Active manager daemon y restarted 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: Activating manager daemon y 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: osdmap e3: 0 total, 0 up, 0 in 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: mgrmap e10: y(active, starting, since 0.00389163s) 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:43:07.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "y", "id": "y"} : dispatch 2024-08-06T17:43:07.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mds metadata"} : dispatch 2024-08-06T17:43:07.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata"} : dispatch 2024-08-06T17:43:07.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata"} : dispatch 2024-08-06T17:43:07.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:07 smithi086 ceph-mon[29229]: Manager daemon y is now available 2024-08-06T17:43:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"} : dispatch 2024-08-06T17:43:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"} : dispatch 2024-08-06T17:43:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:08 smithi086 ceph-mon[29229]: mgrmap e11: y(active, since 1.00573s) 2024-08-06T17:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-mon[29229]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-08-06T17:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-mon[29229]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-08-06T17:43:09.582 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:09.582 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: 2024-08-06T17:43:09.343+0000 7f49ce48d640 -1 log_channel(cephadm) log [ERR] : [06/Aug/2024:17:43:09] ENGINE Error in HTTPServer.serve 2024-08-06T17:43:09.582 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: Traceback (most recent call last): 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._connections.run(self.expiration_interval) 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._run(expiration_interval) 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: new_conn = self._from_server_socket(self.server.socket) 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: s = self.context.wrap_socket( 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: return self.sslsocket_class._create( 2024-08-06T17:43:09.583 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-06T17:43:09.584 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self.do_handshake() 2024-08-06T17:43:09.584 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-06T17:43:09.584 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: self._sslobj.do_handshake() 2024-08-06T17:43:09.584 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-06T17:43:09.584 INFO:journalctl@ceph.mgr.y.smithi086.stdout:Aug 06 17:43:09 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-y[29455]: 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout { 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout "initialized": true 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout } 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:mgr epoch 9 is available 2024-08-06T17:43:09.585 INFO:teuthology.orchestra.run.smithi086.stdout:Using certmgr to generate dashboard self-signed certificate... 2024-08-06T17:43:10.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: [06/Aug/2024:17:43:09] ENGINE Bus STARTING 2024-08-06T17:43:10.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: [06/Aug/2024:17:43:09] ENGINE Serving on http://172.21.15.86:8765 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: [06/Aug/2024:17:43:09] ENGINE Serving on https://172.21.15.86:7150 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: [06/Aug/2024:17:43:09] ENGINE Bus STARTED 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: [06/Aug/2024:17:43:09] ENGINE Error in HTTPServer.serve 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: Traceback (most recent call last): 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._connections.run(self.expiration_interval) 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._run(expiration_interval) 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-08-06T17:43:10.815 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: s = self.context.wrap_socket( 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: return self.sslsocket_class._create( 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: self.do_handshake() 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: self._sslobj.do_handshake() 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: mgrmap e12: y(active, since 2s) 2024-08-06T17:43:10.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:11.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:11 smithi086 ceph-mon[29229]: from='client.14160 -' entity='client.admin' cmd=[{"prefix": "orch certmgr generate-certificates", "module_name": "dashboard", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:11.827 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout {"cert": "-----BEGIN CERTIFICATE-----\nMIIFEDCCAvigAwIBAgIUWSDN/7cBvYxTkQPv/7V/eATpkiQwDQYJKoZIhvcNAQEL\nBQAwFzEVMBMGA1UEAwwMY2VwaGFkbS1yb290MB4XDTI0MDgwNjE3NDMxMVoXDTM0\nMDgwNzE3NDMxMVowFzEVMBMGA1UEAwwMMTcyLjIxLjE1Ljg2MIICIjANBgkqhkiG\n9w0BAQEFAAOCAg8AMIICCgKCAgEAptbCZqLK8QKF/JM4KF5yg900K6p0pWBb8n9H\nmUKJ7s23oPuX1ZXUyPYkz/UqRCKk3k/jn6bJL93dERUg5GfBTlFCr/xRbeD64qgp\nGE5W8xqI1FwW8Ky5UjRBojjF/vPtU+yamQA6fcszrn3+3VpEPlU94dB4dAAJSrW5\n9SQ53mUh0Wqh6rISjgwsnX+CQ93Qe9J95EJMddRhKzH6mjVaH0B6IDjyt2nxgLMn\n1QLzWPiCGoCw63by96uTzGQvDW79Zvtb0vBg43ZoGQ84RI12X53LINgLe+9alYBw\ncFAj3A8L4D5WtK03ljr1Vy3jaScW0CFkMuQyo+l02BWwC5yuuWMY12IxeqT4Oc9X\nrwE+I9jnIXXaSPt30y9GqireFTpLJ23yWIkLM7N1n+La1k8GcU/ChwO10dGe+C2C\n9G75qmvxSAsghc44I/WePqD669+drFd7DrJ+zMNU/2rXE0d/z6qcK7+Dt/6JvpkW\nGu3c1mmYeRA4vOdWoiqcQREBqQBGqWOf5sDSZWpZXjoDQtMBkc55i61OWJQLv8ge\ne1uet+aqd1b2d7/TmoNfcOvqux0QZXk8wXuL0X9NsEwapYU9N4s7Z6eDHD6Cv/Ig\nj6DXQMsy4yRWpZdXFdh1QqC5oQRLzwVUPlCXISS7MmpLf8sYf4GM2m+7WjAdTP3w\nmDM3U1UCAwEAAaNUMFIwQgYDVR0RBDswOYIec21pdGhpMDg2LmZyb250LnNlcGlh\nLmNlcGguY29tghFkYXNoYm9hcmRfc2VydmVyc4cErBUPVjAMBgNVHRMBAf8EAjAA\nMA0GCSqGSIb3DQEBCwUAA4ICAQA9H6TOuZV7SGTckppc07QcNEYk7oKCLPDIHHIv\nbRrUQ4g0IBiUksEcc8FnGnbegigRMvy8fdUOJmCRaGBm36CbKL7w2reMtE9LATsM\nZDgga+iWKNmxvYHLud5B6YdzlpwoLuEU7sOWUmcbSD/+i/MXVX+VNa+AArtDgkmB\ni0KU/oiJzkoFGwgen9OMloKMbCXHazcz0T3MpKIDEJSK0KTZByWmtUSceMocCxFT\nRW5ivIu7HPL5sk5I/mvXHlR4A+LHW4v06Lyd0jE8oJlwvwOeajnaTFbnVWY9JMm5\n8SF4dgpQjggLwWbgpmblPlcJ5pXC3rsgBiHnr5ohS7BuLT0hNqWi3uYxffsjNWax\ndDWRigYHaoQ4SQq8Q1XsNLxzFicYeVbBcllvWFVX6wdDqcK78wiUMkOZugQmuvM8\nLb+MXln2Jrs+jZJZCnHWERnYcJU0zXWfPNP+T+m6HX5edL+mPk0lsUX31XRfPR76\nV0JZo/EPn+QYfQnPDPfPOE5n7QcbVAPRZMovlQ2yj09zgtrdfz3X/CrSLEQPNHnM\nq2z4KweSUt16whOYbrAwdmUuqcLeFJrZsm5bdHvniPJaD8u0DAdpw6KTB0wbLc9O\nGRqQ5n3TYEDK1LXfLOd1FUAr4zwupW04fkc/hMi9Q1XExVi96UfZNqcnsf2xGiJW\n3P7ASw==\n-----END CERTIFICATE-----\n", "key": "-----BEGIN RSA PRIVATE KEY-----\nMIIJKAIBAAKCAgEAptbCZqLK8QKF/JM4KF5yg900K6p0pWBb8n9HmUKJ7s23oPuX\n1ZXUyPYkz/UqRCKk3k/jn6bJL93dERUg5GfBTlFCr/xRbeD64qgpGE5W8xqI1FwW\n8Ky5UjRBojjF/vPtU+yamQA6fcszrn3+3VpEPlU94dB4dAAJSrW59SQ53mUh0Wqh\n6rISjgwsnX+CQ93Qe9J95EJMddRhKzH6mjVaH0B6IDjyt2nxgLMn1QLzWPiCGoCw\n63by96uTzGQvDW79Zvtb0vBg43ZoGQ84RI12X53LINgLe+9alYBwcFAj3A8L4D5W\ntK03ljr1Vy3jaScW0CFkMuQyo+l02BWwC5yuuWMY12IxeqT4Oc9XrwE+I9jnIXXa\nSPt30y9GqireFTpLJ23yWIkLM7N1n+La1k8GcU/ChwO10dGe+C2C9G75qmvxSAsg\nhc44I/WePqD669+drFd7DrJ+zMNU/2rXE0d/z6qcK7+Dt/6JvpkWGu3c1mmYeRA4\nvOdWoiqcQREBqQBGqWOf5sDSZWpZXjoDQtMBkc55i61OWJQLv8gee1uet+aqd1b2\nd7/TmoNfcOvqux0QZXk8wXuL0X9NsEwapYU9N4s7Z6eDHD6Cv/Igj6DXQMsy4yRW\npZdXFdh1QqC5oQRLzwVUPlCXISS7MmpLf8sYf4GM2m+7WjAdTP3wmDM3U1UCAwEA\nAQKCAgArOFBFCWAcXkR6gWLXZlwxF9FHjJFKNDOtltZj+A5GbElpLTQQsw9b/hPc\nQY0w+G6QXs6/UZQlz8pQfhI2yUAU+00UzP8FQXaM1/ZJYyX4NUQEC1UwBPa/OmVU\nUnPMT8GH76+++gWtB8J8LgyjqBossaVlxb0mg8nkbj5dqIRLPhSsLP0EERxoavP1\n2w3Safhz0zwksR4gmiOP7pMWdsLBfNEGXupVpF7ZMdAcSPbkcgA2QGdUoExh57NS\njBN6XenMT0pi0z2uP0LJVx2C+MiVDYoNGJqlAQUM3KiEF+aqP8WaUlvg00ymoOBk\ntLEIUQWCZlwQKPiLaNOJlNi1sWG7nr6ot0LiBozsVmUSk4K3dmroqFt5vndqh/k0\nK9MmnP2iHPSzmIoGwAHkVNi2xQDspO+S0bAhKGCHQyviOS2GlOh3ID9VNw6rNmne\nH7+GREqXmwKOLWP0ITNKN7PUItf/BNwiBP9jb8n8gJp/CjOz4r8YtNJXHw10LJCd\nM0bXzYzcgwtY4mpUChK0yW2aqAOPYY+RkVtcHzJrS6xnqYDKx+aXeAJONnuXNgdO\nfUc7SKomIYt3lKSH+3ldXnHsAC1t6Pl04tl+Os3+SMqtK9Ay9Ctda/HuhacHXI1M\n+jRxnFgf3924bRSdumgpqFzAjZLZUZuPzY9CrR10+haT2pjJgwKCAQEA0haqVvsp\nohK69NyEK+f+EF3dpecRshj3Q2qW2Ln9c90cr8m71TbBs4A1gJYeNy1OFh0SsxFk\nt2QMJZHxjOlLWYs4jMhWzEjNUYNM6COXj1AM8OsUCaYaHIGo/vkC0mijG14zoBg3\nWTthICG+/wjL1TqOMKuM42WpfI5RRqL2M88D2cUfjG5QaBhJW76mwl+5B0O7YxAY\n1M1MoQaLlGDO6Zx4a3jXbGB4GvbGHO0LrD3lP0JSM/+RycUV/QH+G0eDHizU3irW\n+p/cm0N8/pmm+ydporNjxS78nOn/CjYEjQmhhm1kGQDVo/9uAKX1Al6RUXtcFCBE\n3deMhIkIJTpkCwKCAQEAy0yB5K80zMeM/hdNDaqar+Y/PzjPgA8R2i0fKjFD3vG5\nMBjBiUpwoI2V49zBt7m/twIMnzA9LKlCRWZ922zvpAOFKkX5//BQgqEP1xRSopr2\nmYTLn0MORJxxzAgbOoPXY6O1JAzt7TAVIXPPcs8ni/IVEJ3+ZIrY0XKRhvO4h75M\nHX45xO7UhLJyJq2OgeK7fF4QgvxXCQ3QNcMbG8oMSuSsoiqAS1cfZdhe3O2OwGEB\niBmV5vGm2v3ZpndGLUiozT0rPlH9Mv6jonv3S8vCccZnp+64KG3aUK6X2RxY0W8A\n7jY+zZrxLE8byzkJV8mm0/3W+VgN9v22Ks46OytiHwKCAQEAvISKJbMLENwt1ztp\nftyWJqmUEyLdxiqQ9nXXFnMO88Ed1L/5RMH13tTVLaWHR4vn3gAIn5akAcxM+gmT\n7xYI/zjmba339Sf+1X40rLoDbh4u/qBzQpVCKQJQWzunRJeQhSZqvOX+BJw781FU\nyhViktwQa6wEqKi5BYPPessQkjzY/7jbLnBuxnnEmgbiLNkod1RX+fSf05TFhlkR\nNvwHr6kwhX/SDvYh0FBQRfSP3OdNpw4kJXTXpW6s5HO9SU1KYvFnslROxGZe1xQg\nPXYzWHttVwoSJQact1FiAg+m1cgmoTHX+He/PYIfe4Vv7cheLfe8EHNEWKzq8qLk\nJEPZMQKCAQAsQPk9iI4bzYx5eyUAxpRpmdtGrd0taRpfvda83oCBtWi0eY8Xjaa6\nrFdmhJV+ouaCteV9eYi3ghi09ryK2UnJECMasqTa+7rq1JiYHwpl7wmjHW8dugEP\nGmcDHPigYjaur+IseMwKB59x6r3LpiBbrFS9i0bnJ/pOjXv0PnV6xwfWm7WDkuGc\nCSQkGz+cMnjRd8BsYvWnOOSL726lO/Q1EVYq7smCnPJuvwzXVrbuVa95quggn5XM\nsdmBDRGvpdNo6dnR9RMC2L9SDV4ll7M6BRBnU+Wul0sgqQDQ+AS1gk5XjGNbalD5\nbjcEo+POwkyYdPcoy+wuScgvxTiTzLvNAoIBAAppn2c4E0oNsuusB2orH6UojO8s\n/25C3/L0dlBCOA72ajlkqUOOlg+oKge2fHnhffF6XKT/qNriaEuLxuR9pFqWbr+K\n+djqKRzaM3A5y88kRBo3TfNqkivZPvqM/5VHMgA0wSS4pjpheQYV/hb9Nzt3Dbtd\nvrY5rZPtpJ8WTIBlVzBrxXbEoMDmkkbY0OidXx+XLy/PanJNj77ho1IZ9dv8EsQL\n9+8sVlDaFAQReOeAc37SqxiCI0/fGDntt7vl0IkyYXyGFTk4uJdERD+lw+eiwaJO\nA7LO8p8yMqP1a0UEf9Yg8pb51lfJHs9CN7yeyTeLTwkOu5TtmIlS83bAEbA=\n-----END RSA PRIVATE KEY-----\n"} 2024-08-06T17:43:13.312 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout SSL certificate updated 2024-08-06T17:43:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:13 smithi086 ceph-mon[29229]: from='client.14162 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:14.443 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout SSL certificate key updated 2024-08-06T17:43:14.443 INFO:teuthology.orchestra.run.smithi086.stdout:Creating initial admin user... 2024-08-06T17:43:15.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:14 smithi086 ceph-mon[29229]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard set-ssl-certificate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:15.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:15.790 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$D0IS3i6CycwDVSQsD4iTe.4UqIzJ2Mxqizm5pSvIrpU19pSQnzUTS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1722966195, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-08-06T17:43:15.790 INFO:teuthology.orchestra.run.smithi086.stdout:Fetching dashboard port number... 2024-08-06T17:43:16.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:16 smithi086 ceph-mon[29229]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:16.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:16.963 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stdout 8443 2024-08-06T17:43:16.973 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-08-06T17:43:16.973 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout disabled 2024-08-06T17:43:16.980 INFO:teuthology.orchestra.run.smithi086.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-08-06T17:43:16.981 INFO:teuthology.orchestra.run.smithi086.stdout:systemctl: stdout inactive 2024-08-06T17:43:16.981 INFO:teuthology.orchestra.run.smithi086.stdout:firewalld.service is not enabled 2024-08-06T17:43:16.981 INFO:teuthology.orchestra.run.smithi086.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-08-06T17:43:16.982 INFO:teuthology.orchestra.run.smithi086.stdout:Ceph Dashboard is now available at: 2024-08-06T17:43:16.982 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:16.982 INFO:teuthology.orchestra.run.smithi086.stdout: URL: https://smithi086.front.sepia.ceph.com:8443/ 2024-08-06T17:43:16.982 INFO:teuthology.orchestra.run.smithi086.stdout: User: admin 2024-08-06T17:43:16.983 INFO:teuthology.orchestra.run.smithi086.stdout: Password: wbbd59ke9s 2024-08-06T17:43:16.983 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:16.983 INFO:teuthology.orchestra.run.smithi086.stdout:Saving cluster configuration to /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config directory 2024-08-06T17:43:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1523734211' entity='client.admin' cmd={"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} : dispatch 2024-08-06T17:43:18.253 INFO:teuthology.orchestra.run.smithi086.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-08-06T17:43:18.277 INFO:teuthology.orchestra.run.smithi086.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout:Or, if you are only running a single cluster on this host: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout:Please consider enabling telemetry to help improve Ceph: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: ceph telemetry on 2024-08-06T17:43:18.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.279 INFO:teuthology.orchestra.run.smithi086.stdout:For more information see: 2024-08-06T17:43:18.279 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.279 INFO:teuthology.orchestra.run.smithi086.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-08-06T17:43:18.279 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:18.279 INFO:teuthology.orchestra.run.smithi086.stdout:Bootstrap complete. 2024-08-06T17:43:18.308 INFO:tasks.cephadm:Fetching config... 2024-08-06T17:43:18.308 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:43:18.308 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-08-06T17:43:18.324 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-08-06T17:43:18.324 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:43:18.324 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-08-06T17:43:18.380 INFO:tasks.cephadm:Fetching mon keyring... 2024-08-06T17:43:18.381 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:43:18.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/keyring of=/dev/stdout 2024-08-06T17:43:18.445 INFO:tasks.cephadm:Fetching pub ssh key... 2024-08-06T17:43:18.445 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:43:18.445 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-08-06T17:43:18.501 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-08-06T17:43:18.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-06T17:43:18.576 INFO:teuthology.orchestra.run.smithi086.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:43:18.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-06T17:43:18.629 INFO:teuthology.orchestra.run.smithi107.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:43:18.641 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-08-06T17:43:18.682 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCnVQMEdEJ1FJwbhjxJ7/zlQdMhckY3Vmkr6bpcNXcqg3kzZYa8gENPqee8pRAdr0KeTnt0WiX6f/GPIWqIeB2rAxLPeZwPYJVbx+spPtMzzlGob2ak/epeMYMxI2QehRFeWgWfYPIhWj3HsDFn0VA4grsYQO0IehD59SBdGzxyLI2eYjM4OkVI36LmZki81DjVbKNX2gRkXBx88dN6O26DjCKbQhqWBy6rrwYPKf2fnAW9TsDNFUYXPrynuhw2OHagmALZjEwRjH94iqCdps3LnDsvp6Apyx6iTdrf6q1m/AhcOo7cA6R31b8OyZ0BihfHoHcyKAQi4knf71KwJrPmVNzZiRgL/22Lv7XSm1HQXTAk8KychP9shfGYqQEL4U/QVblPJ9Mb0NJvXbCFEmdYe6bVJu0SfqND9ShFsincxuhYXSaKL2qYP6WRY+fcyraPi0e3iLD3CsdC8ZYDkltriic9iraQP5RNyFgqdskHocQzusSnWqjf4KsfRxNHQLs= ceph-fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:43:18.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-08-06T17:43:18.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3304977859' entity='client.admin' 2024-08-06T17:43:18.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:20.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:20 smithi086 ceph-mon[29229]: mgrmap e13: y(active, since 12s) 2024-08-06T17:43:22.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3531830363' entity='client.admin' 2024-08-06T17:43:22.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:22.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:22.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi086", "name": "osd_memory_target"} : dispatch 2024-08-06T17:43:22.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:22.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:22.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:22.764 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-08-06T17:43:22.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-08-06T17:43:23.010 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:25.510 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi107 2024-08-06T17:43:25.510 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:43:25.510 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.conf 2024-08-06T17:43:25.529 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:43:25.529 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:25.588 INFO:tasks.cephadm:Adding host smithi107 to orchestrator... 2024-08-06T17:43:25.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch host add smithi107 2024-08-06T17:43:25.843 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:26.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:26.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:26.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: Updating smithi086:/etc/ceph/ceph.conf 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: Updating smithi086:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.conf 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: Updating smithi086:/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: Updating smithi086:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.client.admin.keyring 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:26.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:28 smithi086 ceph-mon[29229]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi107", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:28 smithi086 ceph-mon[29229]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:29.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:29 smithi086 ceph-mon[29229]: Deploying cephadm binary to smithi107 2024-08-06T17:43:30.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:30 smithi086 ceph-mon[29229]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:30.726 INFO:teuthology.orchestra.run.smithi086.stdout:Added host 'smithi107' with addr '172.21.15.107' 2024-08-06T17:43:31.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch host ls --format=json 2024-08-06T17:43:31.565 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: Added host smithi107 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:32.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:31 smithi086 ceph-mon[29229]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:33.348 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:33.348 INFO:teuthology.orchestra.run.smithi086.stdout:[{"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}] 2024-08-06T17:43:33.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:33 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:33.934 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi117 2024-08-06T17:43:33.934 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:43:33.934 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.conf 2024-08-06T17:43:33.954 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:43:33.954 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:34.013 INFO:tasks.cephadm:Adding host smithi117 to orchestrator... 2024-08-06T17:43:34.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch host add smithi117 2024-08-06T17:43:34.263 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:34 smithi086 ceph-mon[29229]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:43:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:34 smithi086 ceph-mon[29229]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:36.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi107", "name": "osd_memory_target"} : dispatch 2024-08-06T17:43:36.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: Updating smithi107:/etc/ceph/ceph.conf 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: Updating smithi107:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.conf 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: Updating smithi107:/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: Updating smithi107:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.client.admin.keyring 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:36.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:37.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:36 smithi086 ceph-mon[29229]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:38.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:37 smithi086 ceph-mon[29229]: Deploying cephadm binary to smithi117 2024-08-06T17:43:38.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:37 smithi086 ceph-mon[29229]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:39.052 INFO:teuthology.orchestra.run.smithi086.stdout:Added host 'smithi117' with addr '172.21.15.117' 2024-08-06T17:43:39.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch host ls --format=json 2024-08-06T17:43:39.892 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: Added host smithi117 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:40.304 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:41.595 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:43:41.595 INFO:teuthology.orchestra.run.smithi086.stdout:[{"addr": "172.21.15.86", "hostname": "smithi086", "labels": [], "status": ""}, {"addr": "172.21.15.107", "hostname": "smithi107", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}] 2024-08-06T17:43:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:42.152 INFO:tasks.cephadm:Setting crush tunables to default 2024-08-06T17:43:42.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd crush tunables default 2024-08-06T17:43:42.401 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:43:42.522 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:42 smithi086 ceph-mon[29229]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:42.522 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:42 smithi086 ceph-mon[29229]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:43:44.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"} : dispatch 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: Updating smithi117:/etc/ceph/ceph.conf 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: Updating smithi117:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.conf 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:44.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:45.129 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2024-08-06T17:43:45.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:45 smithi086 ceph-mon[29229]: Updating smithi117:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.client.admin.keyring 2024-08-06T17:43:45.447 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1373523470' entity='client.admin' cmd={"prefix": "osd crush tunables", "profile": "default"} : dispatch 2024-08-06T17:43:45.716 INFO:tasks.cephadm:Adding mon.a on smithi086 2024-08-06T17:43:45.716 INFO:tasks.cephadm:Adding mon.b on smithi107 2024-08-06T17:43:45.716 INFO:tasks.cephadm:Adding mon.c on smithi117 2024-08-06T17:43:45.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch apply mon '3;smithi086:172.21.15.86=a;smithi107:172.21.15.107=b;smithi117:172.21.15.117=c' 2024-08-06T17:43:45.967 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.conf 2024-08-06T17:43:46.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1373523470' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-08-06T17:43:46.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:46 smithi086 ceph-mon[29229]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:43:46.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:46 smithi086 ceph-mon[29229]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:47.554 INFO:teuthology.orchestra.run.smithi117.stdout:Scheduled mon update... 2024-08-06T17:43:48.161 DEBUG:teuthology.orchestra.run.smithi107:mon.b> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.b.service 2024-08-06T17:43:48.164 DEBUG:teuthology.orchestra.run.smithi117:mon.c> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.c.service 2024-08-06T17:43:48.166 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-06T17:43:48.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph mon dump -f json 2024-08-06T17:43:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi086:172.21.15.86=a;smithi107:172.21.15.107=b;smithi117:172.21.15.117=c", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:43:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: Saving service mon spec with placement smithi086:172.21.15.86=a;smithi107:172.21.15.107=b;smithi117:172.21.15.117=c;count:3 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:48 smithi086 ceph-mon[29229]: Deploying daemon mon.c on smithi117 2024-08-06T17:43:49.122 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/config/ceph.conf 2024-08-06T17:43:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:50 smithi086 ceph-mon[29229]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:51.303 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:43:51.303 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":1,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","modified":"2024-08-06T17:42:13.412697Z","created":"2024-08-06T17:42:13.412697Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-08-06T17:43:51.304 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 1 2024-08-06T17:43:52.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/1647519494' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:43:52.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:52 smithi086 ceph-mon[29229]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:53 smithi117 ceph-mon[32835]: rocksdb: [db/flush_job.cc:1019] [default] [JOB 3] Flush lasted 4441 microseconds, and 3562 cpu microseconds. 2024-08-06T17:43:53.180 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-06T17:43:53.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph mon dump -f json 2024-08-06T17:43:53.429 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:43:58.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:43:58.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: mon.a calling monitor election 2024-08-06T17:43:58.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: mon.c calling monitor election 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-06T17:43:58.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: monmap epoch 2 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: last_changed 2024-08-06T17:43:53.012712+0000 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: min_mon_release 19 (squid) 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: election_strategy: 1 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: fsmap 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: mgrmap e13: y(active, since 50s) 2024-08-06T17:43:58.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: overall HEALTH_OK 2024-08-06T17:43:58.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:43:58.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: mon.a calling monitor election 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: mon.c calling monitor election 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: monmap epoch 2 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: last_changed 2024-08-06T17:43:53.012712+0000 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: min_mon_release 19 (squid) 2024-08-06T17:43:58.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: election_strategy: 1 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: fsmap 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: mgrmap e13: y(active, since 50s) 2024-08-06T17:43:58.376 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: overall HEALTH_OK 2024-08-06T17:43:58.377 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.377 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.377 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:43:58.377 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:43:58.377 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:43:58.670 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:43:58.670 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":2,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","modified":"2024-08-06T17:43:53.012712Z","created":"2024-08-06T17:42:13.412697Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-06T17:43:58.670 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 2 2024-08-06T17:43:59.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:59 smithi117 ceph-mon[32835]: Deploying daemon mon.b on smithi107 2024-08-06T17:43:59.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/1171704483' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:43:59.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:43:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:43:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:59 smithi086 ceph-mon[29229]: Deploying daemon mon.b on smithi107 2024-08-06T17:43:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/1171704483' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:43:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:43:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:00.285 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-08-06T17:44:00.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph mon dump -f json 2024-08-06T17:44:00.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:00 smithi086 ceph-mon[29229]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:00 smithi117 ceph-mon[32835]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:00.535 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:44:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:01 smithi107 systemd[1]: Started Ceph mon.b for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:44:07.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:07.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: mon.a calling monitor election 2024-08-06T17:44:07.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: mon.c calling monitor election 2024-08-06T17:44:07.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: monmap epoch 3 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:07.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: min_mon_release 19 (squid) 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: election_strategy: 1 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: fsmap 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: mgrmap e13: y(active, since 59s) 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2024-08-06T17:44:07.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.120 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.120 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.120 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: mon.a calling monitor election 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: mon.c calling monitor election 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: monmap epoch 3 2024-08-06T17:44:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: min_mon_release 19 (squid) 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: election_strategy: 1 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: fsmap 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: mgrmap e13: y(active, since 59s) 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-06T17:44:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2024-08-06T17:44:07.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:07.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:08.023 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-08-06T17:44:08.023 INFO:teuthology.orchestra.run.smithi117.stdout:{"epoch":3,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","modified":"2024-08-06T17:44:01.721788Z","created":"2024-08-06T17:42:13.412697Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:3300","nonce":0},{"type":"v1","addr":"172.21.15.86:6789","nonce":0}]},"addr":"172.21.15.86:6789/0","public_addr":"172.21.15.86:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:3300","nonce":0},{"type":"v1","addr":"172.21.15.107:6789","nonce":0}]},"addr":"172.21.15.107:6789/0","public_addr":"172.21.15.107:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-08-06T17:44:08.023 INFO:teuthology.orchestra.run.smithi117.stderr:dumped monmap epoch 3 2024-08-06T17:44:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:07 smithi086 ceph-mon[29229]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:07 smithi117 ceph-mon[32835]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:08.697 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-08-06T17:44:08.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph config generate-minimal-conf 2024-08-06T17:44:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: mon.a calling monitor election 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: mon.c calling monitor election 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: monmap epoch 3 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: min_mon_release 19 (squid) 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: election_strategy: 1 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: fsmap 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: mgrmap e13: y(active, since 59s) 2024-08-06T17:44:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: mon.b (rank 2) addr [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] is down (out of quorum) 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:09.470 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mon.b calling monitor election 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mon.b calling monitor election 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mon.c calling monitor election 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mon.a calling monitor election 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: monmap epoch 3 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: min_mon_release 19 (squid) 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: election_strategy: 1 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: fsmap 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: mgrmap e13: y(active, since 61s) 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: Cluster is now healthy 2024-08-06T17:44:09.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:09 smithi086 ceph-mon[29229]: overall HEALTH_OK 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mon.b calling monitor election 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mon.b calling monitor election 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mon.c calling monitor election 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mon.a calling monitor election 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: monmap epoch 3 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: min_mon_release 19 (squid) 2024-08-06T17:44:09.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: election_strategy: 1 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: fsmap 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: mgrmap e13: y(active, since 61s) 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: Cluster is now healthy 2024-08-06T17:44:09.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:09 smithi117 ceph-mon[32835]: overall HEALTH_OK 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mon.b calling monitor election 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mon.b calling monitor election 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mon.c calling monitor election 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mon.a calling monitor election 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-08-06T17:44:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: monmap epoch 3 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: last_changed 2024-08-06T17:44:01.721788+0000 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: created 2024-08-06T17:42:13.412697+0000 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: min_mon_release 19 (squid) 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: election_strategy: 1 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: 0: [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] mon.a 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: 1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.c 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: 2: [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] mon.b 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: fsmap 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: osdmap e4: 0 total, 0 up, 0 in 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: mgrmap e13: y(active, since 61s) 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: Cluster is now healthy 2024-08-06T17:44:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:09 smithi107 ceph-mon[32381]: overall HEALTH_OK 2024-08-06T17:44:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: Reconfiguring mon.b (monmap changed)... 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:10.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: Reconfiguring mon.b (monmap changed)... 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: Reconfiguring mon.b (monmap changed)... 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.416 INFO:teuthology.orchestra.run.smithi086.stdout:# minimal ceph.conf for fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:11.417 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-08-06T17:44:11.417 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:44:11.417 INFO:teuthology.orchestra.run.smithi086.stdout: mon_host = [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] [v2:172.21.15.107:3300/0,v1:172.21.15.107:6789/0] [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: Reconfiguring daemon mon.b on smithi107 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:11.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3375252415' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: Reconfiguring daemon mon.b on smithi107 2024-08-06T17:44:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:11.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:11.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3375252415' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: Reconfiguring daemon mon.b on smithi107 2024-08-06T17:44:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "mon."} : dispatch 2024-08-06T17:44:11.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config get", "who": "mon", "key": "public_network"} : dispatch 2024-08-06T17:44:11.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:11.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3375252415' entity='client.admin' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:12.010 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-08-06T17:44:12.011 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:44:12.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.conf 2024-08-06T17:44:12.038 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:44:12.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:44:12.103 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:44:12.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.conf 2024-08-06T17:44:12.136 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:44:12.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:44:12.205 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:44:12.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-08-06T17:44:12.233 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:44:12.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-08-06T17:44:12.299 INFO:tasks.cephadm:Adding mgr.y on smithi086 2024-08-06T17:44:12.299 INFO:tasks.cephadm:Adding mgr.x on smithi107 2024-08-06T17:44:12.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch apply mgr '2;smithi086=y;smithi107=x' 2024-08-06T17:44:12.588 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: Reconfiguring mon.c (monmap changed)... 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: Reconfiguring daemon mon.c on smithi117 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: Reconfiguring mon.c (monmap changed)... 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: Reconfiguring daemon mon.c on smithi117 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: Reconfiguring mon.c (monmap changed)... 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: Reconfiguring daemon mon.c on smithi117 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:13.711 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:13.711 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:13.711 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.094 INFO:teuthology.orchestra.run.smithi117.stdout:Scheduled mgr update... 2024-08-06T17:44:14.670 DEBUG:teuthology.orchestra.run.smithi107:mgr.x> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mgr.x.service 2024-08-06T17:44:14.674 DEBUG:tasks.cephadm:set 0 configs 2024-08-06T17:44:14.674 INFO:tasks.cephadm:Deploying OSDs... 2024-08-06T17:44:14.675 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:44:14.675 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-08-06T17:44:14.696 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-06T17:44:14.696 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-08-06T17:44:14.757 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-08-06 17:43:21.379493651 +0000 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-08-06 17:38:06.791666905 +0000 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-08-06 17:38:06.791666905 +0000 2024-08-06T17:44:14.758 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-08-06 17:38:06.791666905 +0000 2024-08-06T17:44:14.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-06T17:44:14.802 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:14.802 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:14.803 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:14.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:14.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-06T17:44:14.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:14.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.821 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-08-06T17:44:14.821 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-08-06T17:44:14.821 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000228524 s, 2.2 MB/s 2024-08-06T17:44:14.823 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:14.881 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-08-06T17:44:14.939 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-06T17:44:14.939 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-08-06T17:44:14.939 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-08-06 17:43:21.412492952 +0000 2024-08-06T17:44:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-08-06 17:38:07.132659441 +0000 2024-08-06T17:44:14.940 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-08-06 17:38:07.132659441 +0000 2024-08-06T17:44:14.940 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-08-06 17:38:07.132659441 +0000 2024-08-06T17:44:14.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-06T17:44:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-08-06T17:44:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-08-06T17:44:15.005 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000234475 s, 2.2 MB/s 2024-08-06T17:44:15.006 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-06T17:44:15.064 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-08-06 17:43:21.446492231 +0000 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-08-06 17:38:07.466652131 +0000 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-08-06 17:38:07.466652131 +0000 2024-08-06T17:44:15.121 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-08-06 17:38:07.466652131 +0000 2024-08-06T17:44:15.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-06T17:44:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-08-06T17:44:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-08-06T17:44:15.185 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000207353 s, 2.5 MB/s 2024-08-06T17:44:15.186 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-06T17:44:15.243 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-08-06 17:43:21.478491552 +0000 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-08-06 17:38:07.803644755 +0000 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-08-06 17:38:07.803644755 +0000 2024-08-06T17:44:15.299 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-08-06 17:38:07.803644755 +0000 2024-08-06T17:44:15.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-06T17:44:15.362 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-08-06T17:44:15.362 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-08-06T17:44:15.362 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000191103 s, 2.7 MB/s 2024-08-06T17:44:15.363 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-06T17:44:15.420 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:44:15.420 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2024-08-06T17:44:15.438 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-06T17:44:15.438 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 674 Links: 1 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-08-06 17:44:15.484568759 +0000 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-08-06 17:38:14.828322448 +0000 2024-08-06T17:44:15.493 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-08-06 17:38:14.828322448 +0000 2024-08-06T17:44:15.494 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-08-06 17:38:14.828322448 +0000 2024-08-06T17:44:15.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-06T17:44:15.555 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:15 smithi107 ceph-mon[32381]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi107=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:15.555 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:15 smithi107 ceph-mon[32381]: Saving service mgr spec with placement smithi086=y;smithi107=x;count:2 2024-08-06T17:44:15.556 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:15 smithi107 ceph-mon[32381]: Deploying daemon mgr.x on smithi107 2024-08-06T17:44:15.712 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-08-06T17:44:15.712 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-08-06T17:44:15.712 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000251467 s, 2.0 MB/s 2024-08-06T17:44:15.714 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-06T17:44:15.771 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 2024-08-06T17:44:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-08-06 17:44:15.671565758 +0000 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-08-06 17:38:15.173316795 +0000 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-08-06 17:38:15.173316795 +0000 2024-08-06T17:44:15.828 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-08-06 17:38:15.173316795 +0000 2024-08-06T17:44:15.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-06T17:44:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:15 smithi086 ceph-mon[29229]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi107=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:15 smithi086 ceph-mon[29229]: Saving service mgr spec with placement smithi086=y;smithi107=x;count:2 2024-08-06T17:44:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:15 smithi086 ceph-mon[29229]: Deploying daemon mgr.x on smithi107 2024-08-06T17:44:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:15 smithi117 ceph-mon[32835]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi086=y;smithi107=x", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:15 smithi117 ceph-mon[32835]: Saving service mgr spec with placement smithi086=y;smithi107=x;count:2 2024-08-06T17:44:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:15 smithi117 ceph-mon[32835]: Deploying daemon mgr.x on smithi107 2024-08-06T17:44:15.889 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:15 smithi107 systemd[1]: Starting Ceph mgr.x for fd7e697e-541a-11ef-bccc-c7b262605968... 2024-08-06T17:44:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-08-06T17:44:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-08-06T17:44:15.892 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000243885 s, 2.1 MB/s 2024-08-06T17:44:15.893 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-06T17:44:15.950 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 711 Links: 1 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-08-06 17:44:15.671565758 +0000 2024-08-06T17:44:16.005 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-08-06 17:38:15.519311127 +0000 2024-08-06T17:44:16.006 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-08-06 17:38:15.519311127 +0000 2024-08-06T17:44:16.006 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-08-06 17:38:15.519311127 +0000 2024-08-06T17:44:16.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-06T17:44:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-08-06T17:44:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-08-06T17:44:16.069 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00020413 s, 2.5 MB/s 2024-08-06T17:44:16.070 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-06T17:44:16.126 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 2024-08-06T17:44:16.169 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:15 smithi107 podman[33614]: 2024-08-06 17:44:15.856563193 +0000 UTC m=+0.016969432 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:44:16.169 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:15 smithi107 podman[33614]: 2024-08-06 17:44:15.976001854 +0000 UTC m=+0.136408072 container create 8d30c4931f1edc66e73a45e74b700be2679786d6375daa9b9a181f766601b9d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, RELEASE=main-93ef537, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-06T17:44:16.169 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:16 smithi107 podman[33614]: 2024-08-06 17:44:16.170862308 +0000 UTC m=+0.331268528 container init 8d30c4931f1edc66e73a45e74b700be2679786d6375daa9b9a181f766601b9d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-x, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 5h/5d Inode: 730 Links: 1 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2024-08-06 17:44:15.671565758 +0000 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2024-08-06 17:38:15.850305704 +0000 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2024-08-06 17:38:15.850305704 +0000 2024-08-06T17:44:16.181 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: 2024-08-06 17:38:15.850305704 +0000 2024-08-06T17:44:16.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-06T17:44:16.244 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2024-08-06T17:44:16.245 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2024-08-06T17:44:16.245 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.00020232 s, 2.5 MB/s 2024-08-06T17:44:16.246 DEBUG:teuthology.orchestra.run.smithi107:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-06T17:44:16.303 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:44:16.303 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-08-06T17:44:16.323 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-08-06T17:44:16.324 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-08-06T17:44:16.382 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-08-06T17:44:16.382 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.382 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-08-06T17:44:16.382 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.382 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.383 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-08-06 17:43:52.448529076 +0000 2024-08-06T17:44:16.383 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-08-06 17:38:06.995788364 +0000 2024-08-06T17:44:16.383 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-08-06 17:38:06.995788364 +0000 2024-08-06T17:44:16.383 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-08-06 17:38:06.995788364 +0000 2024-08-06T17:44:16.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-08-06T17:44:16.450 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-08-06T17:44:16.450 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-08-06T17:44:16.450 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000298461 s, 1.7 MB/s 2024-08-06T17:44:16.452 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-08-06T17:44:16.463 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:16 smithi107 podman[33614]: 2024-08-06 17:44:16.174374854 +0000 UTC m=+0.334781078 container start 8d30c4931f1edc66e73a45e74b700be2679786d6375daa9b9a181f766601b9d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-93ef537, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-06T17:44:16.463 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mgr[33729]: -- 172.21.15.107:0/2983936625 <== mon.2 v2:172.21.15.107:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b09155f0e0 con 0x55b09153d000 2024-08-06T17:44:16.463 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:16 smithi107 bash[33614]: 8d30c4931f1edc66e73a45e74b700be2679786d6375daa9b9a181f766601b9d2 2024-08-06T17:44:16.463 INFO:journalctl@ceph.mgr.x.smithi107.stdout:Aug 06 17:44:16 smithi107 systemd[1]: Started Ceph mgr.x for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:44:16.509 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-08-06 17:43:52.448529076 +0000 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-08-06 17:38:07.327781619 +0000 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-08-06 17:38:07.327781619 +0000 2024-08-06T17:44:16.565 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-08-06 17:38:07.327781619 +0000 2024-08-06T17:44:16.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-08-06T17:44:16.629 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-08-06T17:44:16.629 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-08-06T17:44:16.629 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000221534 s, 2.3 MB/s 2024-08-06T17:44:16.630 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-08-06T17:44:16.687 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-08-06 17:43:52.449529056 +0000 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-08-06 17:38:07.653774997 +0000 2024-08-06T17:44:16.743 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-08-06 17:38:07.653774997 +0000 2024-08-06T17:44:16.744 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-08-06 17:38:07.653774997 +0000 2024-08-06T17:44:16.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-08-06T17:44:16.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:16.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:16.807 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-08-06T17:44:16.807 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-08-06T17:44:16.807 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000216554 s, 2.4 MB/s 2024-08-06T17:44:16.808 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-08-06T17:44:16.865 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:16.919 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 717 Links: 1 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-08-06 17:43:52.449529056 +0000 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-08-06 17:38:07.985768253 +0000 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-08-06 17:38:07.985768253 +0000 2024-08-06T17:44:16.920 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-08-06 17:38:07.985768253 +0000 2024-08-06T17:44:16.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:16.983 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-08-06T17:44:16.983 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-08-06T17:44:16.983 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000217619 s, 2.4 MB/s 2024-08-06T17:44:16.984 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-08-06T17:44:17.041 INFO:tasks.cephadm:Deploying osd.0 on smithi086 with /dev/vg_nvme/lv_4... 2024-08-06T17:44:17.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-08-06T17:44:17.295 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:18.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:18.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:18.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]} : dispatch 2024-08-06T17:44:18.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr services"} : dispatch 2024-08-06T17:44:18.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:18.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:19.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:19 smithi117 ceph-mon[32835]: Reconfiguring mgr.y (unknown last config time)... 2024-08-06T17:44:19.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:19 smithi117 ceph-mon[32835]: Reconfiguring daemon mgr.y on smithi086 2024-08-06T17:44:19.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:19 smithi086 ceph-mon[29229]: Reconfiguring mgr.y (unknown last config time)... 2024-08-06T17:44:19.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:19 smithi086 ceph-mon[29229]: Reconfiguring daemon mgr.y on smithi086 2024-08-06T17:44:19.948 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-08-06T17:44:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:19 smithi107 ceph-mon[32381]: Reconfiguring mgr.y (unknown last config time)... 2024-08-06T17:44:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:19 smithi107 ceph-mon[32381]: Reconfiguring daemon mgr.y on smithi086 2024-08-06T17:44:20.043 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-08-06T17:44:20.056 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-08-06T17:44:20.056 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-08-06T17:44:20.056 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115282 s, 910 MB/s 2024-08-06T17:44:20.057 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-08-06T17:44:20.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:20.642 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_4 2024-08-06T17:44:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.915 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:22.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: Standby manager daemon x started 2024-08-06T17:44:22.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-06T17:44:22.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-06T17:44:22.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-06T17:44:22.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-06T17:44:22.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:22 smithi086 ceph-mon[29229]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: Standby manager daemon x started 2024-08-06T17:44:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-06T17:44:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-06T17:44:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-06T17:44:22.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-06T17:44:22.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:22 smithi117 ceph-mon[32835]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: Standby manager daemon x started 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/crt"} : dispatch 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/crt"} : dispatch 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/x/key"} : dispatch 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: from='mgr.? 172.21.15.107:0/926584210' entity='mgr.x' cmd={"prefix": "config-key get", "key": "mgr/dashboard/key"} : dispatch 2024-08-06T17:44:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:22 smithi107 ceph-mon[32381]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: mgrmap e14: y(active, since 74s), standbys: x 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:23.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: mgrmap e14: y(active, since 74s), standbys: x 2024-08-06T17:44:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-06T17:44:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:23.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: mgrmap e14: y(active, since 74s), standbys: x 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mgr metadata", "who": "x", "id": "x"} : dispatch 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: from='client.14214 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:23.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:24.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:24.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:24 smithi086 ceph-mon[29229]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:24.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:24 smithi117 ceph-mon[32835]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:24 smithi107 ceph-mon[32381]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:25.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"} : dispatch 2024-08-06T17:44:25.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"}]': finished 2024-08-06T17:44:25.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:25 smithi086 ceph-mon[29229]: osdmap e5: 1 total, 0 up, 1 in 2024-08-06T17:44:25.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"} : dispatch 2024-08-06T17:44:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"}]': finished 2024-08-06T17:44:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:25 smithi117 ceph-mon[32835]: osdmap e5: 1 total, 0 up, 1 in 2024-08-06T17:44:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"} : dispatch 2024-08-06T17:44:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3212943321' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2ab66bcc-745f-4928-8cff-7095957437c5"}]': finished 2024-08-06T17:44:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:25 smithi107 ceph-mon[32381]: osdmap e5: 1 total, 0 up, 1 in 2024-08-06T17:44:25.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:26.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/631609470' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:26.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:26 smithi086 ceph-mon[29229]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/631609470' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:26 smithi117 ceph-mon[32835]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/631609470' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:26 smithi107 ceph-mon[32381]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:28.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:28 smithi086 ceph-mon[29229]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:28 smithi117 ceph-mon[32835]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:28 smithi107 ceph-mon[32381]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:30.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:30 smithi086 ceph-mon[29229]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:30 smithi117 ceph-mon[32835]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:30 smithi107 ceph-mon[32381]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-06T17:44:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:31.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:31.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:31.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-06T17:44:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:31.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:31.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:31.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:31.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.0"} : dispatch 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:44:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:32.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:32 smithi086 ceph-mon[29229]: Deploying daemon osd.0 on smithi086 2024-08-06T17:44:32.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:32 smithi086 ceph-mon[29229]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:32 smithi117 ceph-mon[32835]: Deploying daemon osd.0 on smithi086 2024-08-06T17:44:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:32 smithi117 ceph-mon[32835]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:32 smithi107 ceph-mon[32381]: Deploying daemon osd.0 on smithi086 2024-08-06T17:44:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:32 smithi107 ceph-mon[32381]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:34.626 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:34 smithi086 ceph-mon[29229]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:34 smithi117 ceph-mon[32835]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:34 smithi107 ceph-mon[32381]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:36 smithi086 ceph-mon[29229]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:36 smithi117 ceph-mon[32835]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:36 smithi107 ceph-mon[32381]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:38.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:38.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:38.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:38.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:38.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.319 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 0 on host 'smithi086' 2024-08-06T17:44:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:39 smithi086 ceph-mon[29229]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-06T17:44:39.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:39.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:39 smithi117 ceph-mon[32835]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-06T17:44:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:39 smithi107 ceph-mon[32381]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]} : dispatch 2024-08-06T17:44:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:40.009 DEBUG:teuthology.orchestra.run.smithi086:osd.0> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service 2024-08-06T17:44:40.011 INFO:tasks.cephadm:Deploying osd.1 on smithi086 with /dev/vg_nvme/lv_3... 2024-08-06T17:44:40.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-08-06T17:44:40.636 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:44:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:40 smithi117 ceph-mon[32835]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:40 smithi117 ceph-mon[32835]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-06T17:44:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:40 smithi117 ceph-mon[32835]: osdmap e6: 1 total, 0 up, 1 in 2024-08-06T17:44:40.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:40 smithi117 ceph-mon[32835]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:44:40.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:40.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:40 smithi086 ceph-mon[29229]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:40.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:40 smithi086 ceph-mon[29229]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-06T17:44:40.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:40 smithi086 ceph-mon[29229]: osdmap e6: 1 total, 0 up, 1 in 2024-08-06T17:44:40.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:40 smithi086 ceph-mon[29229]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:44:40.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:40 smithi107 ceph-mon[32381]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:40 smithi107 ceph-mon[32381]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-08-06T17:44:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:40 smithi107 ceph-mon[32381]: osdmap e6: 1 total, 0 up, 1 in 2024-08-06T17:44:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:40 smithi107 ceph-mon[32381]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd={"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:44:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-0[43625]: 2024-08-06T17:44:41.045+0000 7f7880728640 -1 osd.0 0 waiting for initial osdmap 2024-08-06T17:44:41.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-0[43625]: 2024-08-06T17:44:41.048+0000 7f7878547640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:44:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: osdmap e7: 1 total, 0 up, 1 in 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: osdmap e7: 1 total, 0 up, 1 in 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' 2024-08-06T17:44:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:41.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:44:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: osdmap e7: 1 total, 0 up, 1 in 2024-08-06T17:44:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: from='osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730]' entity='osd.0' 2024-08-06T17:44:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-08-06T17:44:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730] boot 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: osdmap e8: 1 total, 1 up, 1 in 2024-08-06T17:44:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:44:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:44:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730] boot 2024-08-06T17:44:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: osdmap e8: 1 total, 1 up, 1 in 2024-08-06T17:44:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:43.065 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-08-06T17:44:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: osd.0 [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730] boot 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: osdmap e8: 1 total, 1 up, 1 in 2024-08-06T17:44:43.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 0} : dispatch 2024-08-06T17:44:43.171 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-08-06T17:44:43.188 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-08-06T17:44:43.188 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-08-06T17:44:43.188 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.014889 s, 704 MB/s 2024-08-06T17:44:43.189 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-08-06T17:44:43.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_3 2024-08-06T17:44:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:43 smithi086 ceph-mon[29229]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:43 smithi107 ceph-mon[32381]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:43 smithi117 ceph-mon[32835]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: osdmap e9: 1 total, 1 up, 1 in 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi086 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi086 to 17088M 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:44.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: osdmap e9: 1 total, 1 up, 1 in 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi086 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi086 to 17088M 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: osdmap e9: 1 total, 1 up, 1 in 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi086 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi086 to 17088M 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:44:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:45.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:45 smithi086 ceph-mon[29229]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:45 smithi107 ceph-mon[32381]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:45 smithi117 ceph-mon[32835]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:47 smithi086 ceph-mon[29229]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:47 smithi117 ceph-mon[32835]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:47 smithi107 ceph-mon[32381]: from='client.14241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:44:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:44:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:44:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:48 smithi107 ceph-mon[32381]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:48 smithi086 ceph-mon[29229]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:48 smithi117 ceph-mon[32835]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"} : dispatch 2024-08-06T17:44:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"}]': finished 2024-08-06T17:44:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:49 smithi086 ceph-mon[29229]: osdmap e10: 2 total, 1 up, 2 in 2024-08-06T17:44:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:44:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"} : dispatch 2024-08-06T17:44:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"}]': finished 2024-08-06T17:44:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:49 smithi117 ceph-mon[32835]: osdmap e10: 2 total, 1 up, 2 in 2024-08-06T17:44:49.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:44:49.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"} : dispatch 2024-08-06T17:44:49.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3887259578' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "331fb57c-4513-4398-afda-0ba9442c0a06"}]': finished 2024-08-06T17:44:49.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:49 smithi107 ceph-mon[32381]: osdmap e10: 2 total, 1 up, 2 in 2024-08-06T17:44:49.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:44:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2636467935' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:50 smithi086 ceph-mon[29229]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2636467935' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:50 smithi117 ceph-mon[32835]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2636467935' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:44:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:50 smithi107 ceph-mon[32381]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:52 smithi086 ceph-mon[29229]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:52 smithi117 ceph-mon[32835]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:52.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:52 smithi107 ceph-mon[32381]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:54 smithi086 ceph-mon[29229]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-06T17:44:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:54 smithi117 ceph-mon[32835]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-06T17:44:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:54 smithi107 ceph-mon[32381]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.1"} : dispatch 2024-08-06T17:44:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:44:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:55 smithi086 ceph-mon[29229]: Deploying daemon osd.1 on smithi086 2024-08-06T17:44:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:55 smithi117 ceph-mon[32835]: Deploying daemon osd.1 on smithi086 2024-08-06T17:44:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:55 smithi107 ceph-mon[32381]: Deploying daemon osd.1 on smithi086 2024-08-06T17:44:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:56 smithi086 ceph-mon[29229]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:56 smithi117 ceph-mon[32835]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:56 smithi107 ceph-mon[32381]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:58 smithi086 ceph-mon[29229]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:58 smithi117 ceph-mon[32835]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:58 smithi107 ceph-mon[32381]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:44:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:44:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:44:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:44:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:44:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:44:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:00.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:00 smithi086 ceph-mon[29229]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:00 smithi117 ceph-mon[32835]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:00 smithi107 ceph-mon[32381]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:02.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:02.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:02.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:02.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.785 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 1 on host 'smithi086' 2024-08-06T17:45:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:02.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.415 DEBUG:teuthology.orchestra.run.smithi086:osd.1> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service 2024-08-06T17:45:03.418 INFO:tasks.cephadm:Deploying osd.2 on smithi086 with /dev/vg_nvme/lv_2... 2024-08-06T17:45:03.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-08-06T17:45:03.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:03 smithi086 ceph-mon[29229]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-06T17:45:03.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:03.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.670 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:03 smithi117 ceph-mon[32835]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-06T17:45:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:03 smithi107 ceph-mon[32381]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]} : dispatch 2024-08-06T17:45:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:04.581 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:45:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-mon[29229]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-mon[29229]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-06T17:45:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-mon[29229]: osdmap e11: 2 total, 1 up, 2 in 2024-08-06T17:45:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-mon[29229]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:04.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:04.867 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-1[49377]: 2024-08-06T17:45:04.608+0000 7fdd56337640 -1 osd.1 0 waiting for initial osdmap 2024-08-06T17:45:04.867 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:04 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-1[49377]: 2024-08-06T17:45:04.611+0000 7fdd5215e640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:45:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:04 smithi117 ceph-mon[32835]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:04 smithi117 ceph-mon[32835]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-06T17:45:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:04 smithi117 ceph-mon[32835]: osdmap e11: 2 total, 1 up, 2 in 2024-08-06T17:45:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:04 smithi117 ceph-mon[32835]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:04 smithi107 ceph-mon[32381]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:04 smithi107 ceph-mon[32381]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-08-06T17:45:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:04 smithi107 ceph-mon[32381]: osdmap e11: 2 total, 1 up, 2 in 2024-08-06T17:45:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:04 smithi107 ceph-mon[32381]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd={"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:45:05.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:45:05.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:05.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:05.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: osdmap e12: 2 total, 1 up, 2 in 2024-08-06T17:45:05.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:45:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:45:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: osdmap e12: 2 total, 1 up, 2 in 2024-08-06T17:45:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: from='osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: osdmap e12: 2 total, 1 up, 2 in 2024-08-06T17:45:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:05.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100] boot 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: osdmap e13: 2 total, 2 up, 2 in 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100] boot 2024-08-06T17:45:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: osdmap e13: 2 total, 2 up, 2 in 2024-08-06T17:45:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: osd.1 [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100] boot 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: osdmap e13: 2 total, 2 up, 2 in 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 1} : dispatch 2024-08-06T17:45:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:07.015 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-08-06T17:45:07.107 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-08-06T17:45:07.123 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-08-06T17:45:07.123 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-08-06T17:45:07.123 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135492 s, 774 MB/s 2024-08-06T17:45:07.123 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: osdmap e14: 2 total, 2 up, 2 in 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi086 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi086 to 8544M 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:07 smithi086 ceph-mon[29229]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:07.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_2 2024-08-06T17:45:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: osdmap e14: 2 total, 2 up, 2 in 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi086 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi086 to 8544M 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:07.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:07.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:07.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:07 smithi107 ceph-mon[32381]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:08.086 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:45:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: osdmap e14: 2 total, 2 up, 2 in 2024-08-06T17:45:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true} : dispatch 2024-08-06T17:45:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi086 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi086 to 8544M 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:07 smithi117 ceph-mon[32835]: pgmap v64: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 ceph-mon[29229]: osdmap e15: 2 total, 2 up, 2 in 2024-08-06T17:45:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 ceph-mon[29229]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-06T17:45:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 ceph-mon[29229]: osdmap e16: 2 total, 2 up, 2 in 2024-08-06T17:45:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:08 smithi107 ceph-mon[32381]: osdmap e15: 2 total, 2 up, 2 in 2024-08-06T17:45:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:08 smithi107 ceph-mon[32381]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-06T17:45:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:08 smithi107 ceph-mon[32381]: osdmap e16: 2 total, 2 up, 2 in 2024-08-06T17:45:09.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-08-06T17:45:09.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:08 smithi117 ceph-mon[32835]: osdmap e15: 2 total, 2 up, 2 in 2024-08-06T17:45:09.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:08 smithi117 ceph-mon[32835]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-08-06T17:45:09.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:08 smithi117 ceph-mon[32835]: osdmap e16: 2 total, 2 up, 2 in 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53104]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53104]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53104]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53104]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 sudo[53107]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 sudo[53107]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 sudo[53107]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 sudo[53107]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53092]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53092]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53092]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53092]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53095]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53095]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53095]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53095]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53098]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53098]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53098]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53098]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53101]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53101]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.189 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53101]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.190 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:45:08 smithi086 sudo[53101]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34800]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34800]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34800]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34800]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34803]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34803]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34803]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 sudo[34803]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34432]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34432]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34432]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34432]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34435]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34435]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-08-06T17:45:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34435]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-08-06T17:45:09.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 sudo[34435]: pam_unix(sudo:session): session closed for user root 2024-08-06T17:45:10.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: osdmap e17: 2 total, 2 up, 2 in 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:10.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: osdmap e17: 2 total, 2 up, 2 in 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:10.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:10.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: pgmap v67: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: osdmap e17: 2 total, 2 up, 2 in 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:10.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:10 smithi107 ceph-mon[32381]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:10 smithi107 ceph-mon[32381]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-06T17:45:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:10 smithi107 ceph-mon[32381]: Cluster is now healthy 2024-08-06T17:45:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:10 smithi107 ceph-mon[32381]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-06T17:45:11.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:10 smithi086 ceph-mon[29229]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:11.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:10 smithi086 ceph-mon[29229]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-06T17:45:11.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:10 smithi086 ceph-mon[29229]: Cluster is now healthy 2024-08-06T17:45:11.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:10 smithi086 ceph-mon[29229]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-06T17:45:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:10 smithi117 ceph-mon[32835]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:10 smithi117 ceph-mon[32835]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-08-06T17:45:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:10 smithi117 ceph-mon[32835]: Cluster is now healthy 2024-08-06T17:45:11.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:10 smithi117 ceph-mon[32835]: mgrmap e15: y(active, since 2m), standbys: x 2024-08-06T17:45:12.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:11 smithi107 ceph-mon[32381]: pgmap v69: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:11 smithi086 ceph-mon[29229]: pgmap v69: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:11 smithi117 ceph-mon[32835]: pgmap v69: 1 pgs: 1 unknown; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:13.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"} : dispatch 2024-08-06T17:45:13.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"}]': finished 2024-08-06T17:45:13.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:12 smithi086 ceph-mon[29229]: osdmap e18: 3 total, 2 up, 3 in 2024-08-06T17:45:13.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"} : dispatch 2024-08-06T17:45:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"}]': finished 2024-08-06T17:45:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:12 smithi107 ceph-mon[32381]: osdmap e18: 3 total, 2 up, 3 in 2024-08-06T17:45:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"} : dispatch 2024-08-06T17:45:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2151354398' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "51280800-811c-42ce-97c0-aa3787f5a681"}]': finished 2024-08-06T17:45:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:12 smithi117 ceph-mon[32835]: osdmap e18: 3 total, 2 up, 3 in 2024-08-06T17:45:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2129412971' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:13 smithi107 ceph-mon[32381]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:14.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2129412971' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:14.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:13 smithi086 ceph-mon[29229]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:14.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2129412971' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:14.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:13 smithi117 ceph-mon[32835]: pgmap v71: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:16.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:16 smithi086 ceph-mon[29229]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:16 smithi117 ceph-mon[32835]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:16 smithi107 ceph-mon[32381]: pgmap v72: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:18 smithi086 ceph-mon[29229]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-06T17:45:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:18 smithi117 ceph-mon[32835]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-06T17:45:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:18 smithi107 ceph-mon[32381]: pgmap v73: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.2"} : dispatch 2024-08-06T17:45:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:19.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:19 smithi086 ceph-mon[29229]: Deploying daemon osd.2 on smithi086 2024-08-06T17:45:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:19 smithi117 ceph-mon[32835]: Deploying daemon osd.2 on smithi086 2024-08-06T17:45:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:19 smithi107 ceph-mon[32381]: Deploying daemon osd.2 on smithi086 2024-08-06T17:45:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:20 smithi086 ceph-mon[29229]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:20 smithi117 ceph-mon[32835]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:20 smithi107 ceph-mon[32381]: pgmap v74: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:22.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:22 smithi086 ceph-mon[29229]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:22 smithi117 ceph-mon[32835]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:22 smithi107 ceph-mon[32381]: pgmap v75: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:23.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:24 smithi086 ceph-mon[29229]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:24 smithi117 ceph-mon[32835]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:24 smithi107 ceph-mon[32381]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:26.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:26.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.759 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 2 on host 'smithi086' 2024-08-06T17:45:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-06T17:45:26.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:26.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:26.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]} : dispatch 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.384 DEBUG:teuthology.orchestra.run.smithi086:osd.2> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service 2024-08-06T17:45:27.387 INFO:tasks.cephadm:Deploying osd.3 on smithi086 with /dev/vg_nvme/lv_1... 2024-08-06T17:45:27.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-08-06T17:45:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-06T17:45:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: osdmap e19: 3 total, 2 up, 3 in 2024-08-06T17:45:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-06T17:45:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: osdmap e19: 3 total, 2 up, 3 in 2024-08-06T17:45:27.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:27.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: osdmap e19: 3 total, 2 up, 3 in 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd={"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.573 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-2[55035]: 2024-08-06T17:45:28.526+0000 7f1d5ef6e640 -1 osd.2 0 waiting for initial osdmap 2024-08-06T17:45:28.574 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-2[55035]: 2024-08-06T17:45:28.530+0000 7f1d56d8d640 -1 osd.2 20 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:45:28.630 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: osdmap e20: 3 total, 2 up, 3 in 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:28 smithi086 ceph-mon[29229]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' 2024-08-06T17:45:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: osdmap e20: 3 total, 2 up, 3 in 2024-08-06T17:45:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:28 smithi117 ceph-mon[32835]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: osdmap e20: 3 total, 2 up, 3 in 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:28 smithi107 ceph-mon[32381]: from='osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493]' entity='osd.2' 2024-08-06T17:45:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:45:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:45:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493] boot 2024-08-06T17:45:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: osdmap e21: 3 total, 3 up, 3 in 2024-08-06T17:45:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493] boot 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: osdmap e21: 3 total, 3 up, 3 in 2024-08-06T17:45:29.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:45:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:45:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:29.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:29.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: osd.2 [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493] boot 2024-08-06T17:45:29.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: osdmap e21: 3 total, 3 up, 3 in 2024-08-06T17:45:29.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 2} : dispatch 2024-08-06T17:45:30.613 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-08-06T17:45:30.700 INFO:teuthology.orchestra.run.smithi086.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-08-06T17:45:30.715 INFO:teuthology.orchestra.run.smithi086.stderr: stderr: 10+0 records in 2024-08-06T17:45:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:10+0 records out 2024-08-06T17:45:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0133659 s, 785 MB/s 2024-08-06T17:45:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:--> Zapping successful for: 2024-08-06T17:45:30.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:30 smithi086 ceph-mon[29229]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:30 smithi117 ceph-mon[32835]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:30 smithi107 ceph-mon[32381]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 54 MiB used, 179 GiB / 179 GiB avail 2024-08-06T17:45:31.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi086:vg_nvme/lv_1 2024-08-06T17:45:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:31 smithi086 ceph-mon[29229]: osdmap e22: 3 total, 3 up, 3 in 2024-08-06T17:45:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:31 smithi086 ceph-mon[29229]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:45:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:31 smithi107 ceph-mon[32381]: osdmap e22: 3 total, 3 up, 3 in 2024-08-06T17:45:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:31 smithi107 ceph-mon[32381]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:31 smithi117 ceph-mon[32835]: osdmap e22: 3 total, 3 up, 3 in 2024-08-06T17:45:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:31 smithi117 ceph-mon[32835]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:33.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi086 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi086 to 5696M 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:33.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi086 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi086 to 5696M 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi086 2024-08-06T17:45:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi086 to 5696M 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:33.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:34.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:33 smithi086 ceph-mon[29229]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:34.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:33 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:34.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:33 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:34.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:33 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:33 smithi117 ceph-mon[32835]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:33 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:34.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:33 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:34.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:33 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:33 smithi107 ceph-mon[32381]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:33 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:33 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:33 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:35.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:34 smithi086 ceph-mon[29229]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:34 smithi117 ceph-mon[32835]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:35.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:34 smithi107 ceph-mon[32381]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi086:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:35 smithi086 ceph-mon[29229]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:35 smithi117 ceph-mon[32835]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:36.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:35 smithi107 ceph-mon[32381]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"} : dispatch 2024-08-06T17:45:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"}]': finished 2024-08-06T17:45:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:36 smithi086 ceph-mon[29229]: osdmap e23: 4 total, 3 up, 4 in 2024-08-06T17:45:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/614951453' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"} : dispatch 2024-08-06T17:45:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"}]': finished 2024-08-06T17:45:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:36 smithi117 ceph-mon[32835]: osdmap e23: 4 total, 3 up, 4 in 2024-08-06T17:45:37.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:37.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/614951453' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:37.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"} : dispatch 2024-08-06T17:45:37.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2708361024' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "121c9b13-fa41-4680-8424-c96f1ddcec09"}]': finished 2024-08-06T17:45:37.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:36 smithi107 ceph-mon[32381]: osdmap e23: 4 total, 3 up, 4 in 2024-08-06T17:45:37.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:37.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/614951453' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:38.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:37 smithi086 ceph-mon[29229]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:37 smithi117 ceph-mon[32835]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:38.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:37 smithi107 ceph-mon[32381]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:40.701 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:40 smithi086 ceph-mon[29229]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:40 smithi117 ceph-mon[32835]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:40 smithi107 ceph-mon[32381]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:42.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:42 smithi086 ceph-mon[29229]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:42.779 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-06T17:45:42.779 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:42 smithi117 ceph-mon[32835]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-06T17:45:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:42 smithi107 ceph-mon[32381]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.3"} : dispatch 2024-08-06T17:45:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:43.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:43 smithi086 ceph-mon[29229]: Deploying daemon osd.3 on smithi086 2024-08-06T17:45:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:43 smithi117 ceph-mon[32835]: Deploying daemon osd.3 on smithi086 2024-08-06T17:45:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:43 smithi107 ceph-mon[32381]: Deploying daemon osd.3 on smithi086 2024-08-06T17:45:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:44 smithi086 ceph-mon[29229]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:44 smithi117 ceph-mon[32835]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:44 smithi107 ceph-mon[32381]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:46.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:46 smithi086 ceph-mon[29229]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:46 smithi117 ceph-mon[32835]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:46 smithi107 ceph-mon[32381]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:47.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:47.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:47.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:48 smithi086 ceph-mon[29229]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:48 smithi117 ceph-mon[32835]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:48 smithi107 ceph-mon[32381]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:49.714 INFO:teuthology.orchestra.run.smithi086.stdout:Created osd(s) 3 on host 'smithi086' 2024-08-06T17:45:50.317 DEBUG:teuthology.orchestra.run.smithi086:osd.3> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service 2024-08-06T17:45:50.320 INFO:tasks.cephadm:Deploying osd.4 on smithi107 with /dev/vg_nvme/lv_4... 2024-08-06T17:45:50.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-08-06T17:45:50.581 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:45:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:50 smithi117 ceph-mon[32835]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:50 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:45:50.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:50 smithi107 ceph-mon[32381]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:50.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:50.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:50.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:50 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:45:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:50 smithi086 ceph-mon[29229]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:50 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:45:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:45:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: osdmap e24: 4 total, 3 up, 4 in 2024-08-06T17:45:51.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:51.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:51.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:51 smithi086 ceph-mon[29229]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:45:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: osdmap e24: 4 total, 3 up, 4 in 2024-08-06T17:45:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:51.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:51 smithi107 ceph-mon[32381]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:45:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: osdmap e24: 4 total, 3 up, 4 in 2024-08-06T17:45:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:45:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:51 smithi117 ceph-mon[32835]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:52.293 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-08-06T17:45:52.405 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-08-06T17:45:52.418 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-08-06T17:45:52.419 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-08-06T17:45:52.419 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114375 s, 917 MB/s 2024-08-06T17:45:52.419 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-08-06T17:45:52.491 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[60694]: 2024-08-06T17:45:52.157+0000 7f2b4bac7640 -1 osd.3 0 waiting for initial osdmap 2024-08-06T17:45:52.491 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[60694]: 2024-08-06T17:45:52.161+0000 7f2b438e6640 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:45:52.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:52.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: osdmap e25: 4 total, 3 up, 4 in 2024-08-06T17:45:52.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:52.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:52.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' 2024-08-06T17:45:52.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.000 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:53.000 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: osdmap e25: 4 total, 3 up, 4 in 2024-08-06T17:45:53.000 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' 2024-08-06T17:45:53.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_4 2024-08-06T17:45:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]}]': finished 2024-08-06T17:45:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: osdmap e25: 4 total, 3 up, 4 in 2024-08-06T17:45:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214]' entity='osd.3' 2024-08-06T17:45:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.256 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214] boot 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: osdmap e26: 4 total, 4 up, 4 in 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:53 smithi107 ceph-mon[32381]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:45:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:45:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214] boot 2024-08-06T17:45:54.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: osdmap e26: 4 total, 4 up, 4 in 2024-08-06T17:45:54.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:54.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:53 smithi086 ceph-mon[29229]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:45:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: osd.3 [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214] boot 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: osdmap e26: 4 total, 4 up, 4 in 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:45:54.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:53 smithi117 ceph-mon[32835]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 81 MiB used, 268 GiB / 268 GiB avail 2024-08-06T17:45:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi086 2024-08-06T17:45:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi086 to 4272M 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:54.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:54.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:54 smithi107 ceph-mon[32381]: osdmap e27: 4 total, 4 up, 4 in 2024-08-06T17:45:55.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi086 2024-08-06T17:45:55.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi086 to 4272M 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:55.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:54 smithi086 ceph-mon[29229]: osdmap e27: 4 total, 4 up, 4 in 2024-08-06T17:45:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi086 2024-08-06T17:45:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi086 to 4272M 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:45:55.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:54 smithi117 ceph-mon[32835]: osdmap e27: 4 total, 4 up, 4 in 2024-08-06T17:45:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:55 smithi107 ceph-mon[32381]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:55 smithi107 ceph-mon[32381]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:56.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:55 smithi086 ceph-mon[29229]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:56.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:56.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:56.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:56.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:55 smithi086 ceph-mon[29229]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:55 smithi117 ceph-mon[32835]: from='client.24182 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:45:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:45:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:45:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:45:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:55 smithi117 ceph-mon[32835]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/2508053473' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:57 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:57 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"}]': finished 2024-08-06T17:45:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:57 smithi086 ceph-mon[29229]: osdmap e28: 5 total, 4 up, 5 in 2024-08-06T17:45:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:45:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/2508053473' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:57 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:57 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"}]': finished 2024-08-06T17:45:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:57 smithi117 ceph-mon[32835]: osdmap e28: 5 total, 4 up, 5 in 2024-08-06T17:45:57.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:45:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/2508053473' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:57 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"} : dispatch 2024-08-06T17:45:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:57 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1dd2693-1793-4b19-84da-e020a8252b4d"}]': finished 2024-08-06T17:45:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:57 smithi107 ceph-mon[32381]: osdmap e28: 5 total, 4 up, 5 in 2024-08-06T17:45:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:45:58.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:58 smithi086 ceph-mon[29229]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:58.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:45:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/1495530644' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:58 smithi117 ceph-mon[32835]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:45:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/1495530644' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:45:58.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:58 smithi107 ceph-mon[32381]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:45:58.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:45:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/1495530644' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:00.606 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:00 smithi107 ceph-mon[32381]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:00.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:00 smithi086 ceph-mon[29229]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:00 smithi117 ceph-mon[32835]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:02.769 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:02 smithi107 ceph-mon[32381]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:02 smithi086 ceph-mon[29229]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:02 smithi117 ceph-mon[32835]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:03.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:03.733 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:03.733 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-06T17:46:03.733 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:46:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-06T17:46:03.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:46:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:46:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.4"} : dispatch 2024-08-06T17:46:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:46:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:46:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: Deploying daemon osd.4 on smithi107 2024-08-06T17:46:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:04.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: osdmap e29: 5 total, 4 up, 5 in 2024-08-06T17:46:04.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:46:04.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:46:04.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: Deploying daemon osd.4 on smithi107 2024-08-06T17:46:04.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:04.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: osdmap e29: 5 total, 4 up, 5 in 2024-08-06T17:46:04.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: Deploying daemon osd.4 on smithi107 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: osdmap e29: 5 total, 4 up, 5 in 2024-08-06T17:46:04.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:06.777 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:06 smithi107 ceph-mon[32381]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:06.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:06 smithi086 ceph-mon[29229]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:06 smithi117 ceph-mon[32835]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:08.690 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:08 smithi107 ceph-mon[32381]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:08.690 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:08.690 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:08.690 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:08 smithi117 ceph-mon[32835]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:08.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:08 smithi086 ceph-mon[29229]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:08.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:08.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:08.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:10.735 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:10 smithi107 ceph-mon[32381]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:10.777 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 4 on host 'smithi107' 2024-08-06T17:46:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:10 smithi117 ceph-mon[32835]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:11.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:10 smithi086 ceph-mon[29229]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:11.644 DEBUG:teuthology.orchestra.run.smithi107:osd.4> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service 2024-08-06T17:46:11.647 INFO:tasks.cephadm:Deploying osd.5 on smithi107 with /dev/vg_nvme/lv_3... 2024-08-06T17:46:11.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-08-06T17:46:11.945 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:11 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:12.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:12.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:12.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:11 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]} : dispatch 2024-08-06T17:46:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:12.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:12.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:11 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:46:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:46:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-06T17:46:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: osdmap e30: 5 total, 4 up, 5 in 2024-08-06T17:46:13.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:13.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-mon[32381]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:13.214 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4[36932]: 2024-08-06T17:46:13.126+0000 7f336ba2e640 -1 osd.4 0 waiting for initial osdmap 2024-08-06T17:46:13.214 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:46:13 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4[36932]: 2024-08-06T17:46:13.131+0000 7f3367855640 -1 osd.4 31 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: osdmap e30: 5 total, 4 up, 5 in 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:13.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:13 smithi117 ceph-mon[32835]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:46:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:46:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-08-06T17:46:13.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: from='osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981]' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:13.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: osdmap e30: 5 total, 4 up, 5 in 2024-08-06T17:46:13.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:13.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:13 smithi086 ceph-mon[29229]: from='osd.4 ' entity='osd.4' cmd={"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:46:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: osdmap e31: 5 total, 4 up, 5 in 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:14 smithi107 ceph-mon[32381]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:46:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:46:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:14.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: osdmap e31: 5 total, 4 up, 5 in 2024-08-06T17:46:14.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:14.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:14 smithi086 ceph-mon[29229]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: osdmap e31: 5 total, 4 up, 5 in 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:14.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:14 smithi117 ceph-mon[32835]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-08-06T17:46:15.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:15 smithi107 ceph-mon[32381]: osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981] boot 2024-08-06T17:46:15.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:15 smithi107 ceph-mon[32381]: osdmap e32: 5 total, 5 up, 5 in 2024-08-06T17:46:15.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:15 smithi086 ceph-mon[29229]: osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981] boot 2024-08-06T17:46:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:15 smithi086 ceph-mon[29229]: osdmap e32: 5 total, 5 up, 5 in 2024-08-06T17:46:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:15 smithi117 ceph-mon[32835]: osd.4 [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981] boot 2024-08-06T17:46:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:15 smithi117 ceph-mon[32835]: osdmap e32: 5 total, 5 up, 5 in 2024-08-06T17:46:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 4} : dispatch 2024-08-06T17:46:15.802 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-08-06T17:46:15.880 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-08-06T17:46:15.897 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-08-06T17:46:15.897 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-08-06T17:46:15.897 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0147882 s, 709 MB/s 2024-08-06T17:46:15.898 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-08-06T17:46:16.448 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: osdmap e33: 5 total, 5 up, 5 in 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi107 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi107 to 17087M 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.450 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:16 smithi107 ceph-mon[32381]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:16.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_3 2024-08-06T17:46:16.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: osdmap e33: 5 total, 5 up, 5 in 2024-08-06T17:46:16.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi107 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi107 to 17087M 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:16 smithi086 ceph-mon[29229]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: osdmap e33: 5 total, 5 up, 5 in 2024-08-06T17:46:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi107 2024-08-06T17:46:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi107 to 17087M 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:16 smithi117 ceph-mon[32835]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:16.729 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:46:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:18 smithi086 ceph-mon[29229]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:18 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:18 smithi117 ceph-mon[32835]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:18 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:18.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:18 smithi107 ceph-mon[32381]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:18.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:18.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:18.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:18 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:19 smithi086 ceph-mon[29229]: 1.0 scrub starts 2024-08-06T17:46:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:19 smithi086 ceph-mon[29229]: 1.0 scrub ok 2024-08-06T17:46:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:19 smithi086 ceph-mon[29229]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:19 smithi117 ceph-mon[32835]: 1.0 scrub starts 2024-08-06T17:46:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:19 smithi117 ceph-mon[32835]: 1.0 scrub ok 2024-08-06T17:46:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:19 smithi117 ceph-mon[32835]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:19.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:19 smithi107 ceph-mon[32381]: 1.0 scrub starts 2024-08-06T17:46:19.955 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:19 smithi107 ceph-mon[32381]: 1.0 scrub ok 2024-08-06T17:46:19.955 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:19 smithi107 ceph-mon[32381]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:20 smithi086 ceph-mon[29229]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:20 smithi117 ceph-mon[32835]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:20 smithi107 ceph-mon[32381]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/1487636986' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"}]': finished 2024-08-06T17:46:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: osdmap e34: 6 total, 5 up, 6 in 2024-08-06T17:46:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/3345912717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/1487636986' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"}]': finished 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: osdmap e34: 6 total, 5 up, 6 in 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/3345912717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/1487636986' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"} : dispatch 2024-08-06T17:46:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "27dd75cf-526a-44b5-8f5e-c53573212054"}]': finished 2024-08-06T17:46:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: osdmap e34: 6 total, 5 up, 6 in 2024-08-06T17:46:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:21.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/3345912717' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:22 smithi117 ceph-mon[32835]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:22 smithi107 ceph-mon[32381]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:22 smithi086 ceph-mon[29229]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:24 smithi117 ceph-mon[32835]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:24.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:24 smithi107 ceph-mon[32381]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:24 smithi086 ceph-mon[29229]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:26.807 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:26 smithi107 ceph-mon[32381]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:26 smithi117 ceph-mon[32835]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:26 smithi086 ceph-mon[29229]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-06T17:46:28.007 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:28.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-06T17:46:28.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.5"} : dispatch 2024-08-06T17:46:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:28.665 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:28 smithi107 ceph-mon[32381]: Deploying daemon osd.5 on smithi107 2024-08-06T17:46:28.665 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:28 smithi107 ceph-mon[32381]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:28 smithi086 ceph-mon[29229]: Deploying daemon osd.5 on smithi107 2024-08-06T17:46:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:28 smithi086 ceph-mon[29229]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:28 smithi117 ceph-mon[32835]: Deploying daemon osd.5 on smithi107 2024-08-06T17:46:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:28 smithi117 ceph-mon[32835]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:29 smithi107 ceph-mon[32381]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:29 smithi086 ceph-mon[29229]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:29 smithi117 ceph-mon[32835]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:46:32.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:32 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:46:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:46:32.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:32 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:46:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:46:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:32 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:33 smithi107 ceph-mon[32381]: osdmap e35: 6 total, 5 up, 6 in 2024-08-06T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:33 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:34.011 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:33 smithi107 ceph-mon[32381]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:33 smithi086 ceph-mon[29229]: osdmap e35: 6 total, 5 up, 6 in 2024-08-06T17:46:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:33 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:33 smithi086 ceph-mon[29229]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:33 smithi117 ceph-mon[32835]: osdmap e35: 6 total, 5 up, 6 in 2024-08-06T17:46:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:33 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:33 smithi117 ceph-mon[32835]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:35 smithi107 ceph-mon[32381]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.243 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 5 on host 'smithi107' 2024-08-06T17:46:35.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:35.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:35.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:35 smithi086 ceph-mon[29229]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:35 smithi117 ceph-mon[32835]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]} : dispatch 2024-08-06T17:46:35.817 DEBUG:teuthology.orchestra.run.smithi107:osd.5> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service 2024-08-06T17:46:35.819 INFO:tasks.cephadm:Deploying osd.6 on smithi107 with /dev/vg_nvme/lv_2... 2024-08-06T17:46:35.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-08-06T17:46:36.103 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-06T17:46:36.103 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: osdmap e36: 6 total, 5 up, 6 in 2024-08-06T17:46:36.103 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:36.103 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.103 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.104 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:36.104 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.104 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.104 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-mon[32381]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:36.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-06T17:46:36.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: osdmap e36: 6 total, 5 up, 6 in 2024-08-06T17:46:36.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:36.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:36.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:36 smithi086 ceph-mon[29229]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-08-06T17:46:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: osdmap e36: 6 total, 5 up, 6 in 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119]' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='osd.5 ' entity='osd.5' cmd={"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:36.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:36 smithi117 ceph-mon[32835]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:36.578 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:46:36.779 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-5[42803]: 2024-08-06T17:46:36.471+0000 7f227016d640 -1 osd.5 0 waiting for initial osdmap 2024-08-06T17:46:36.779 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:46:36 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-5[42803]: 2024-08-06T17:46:36.476+0000 7f2267f8c640 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:46:37.348 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:37.348 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: osdmap e37: 6 total, 5 up, 6 in 2024-08-06T17:46:37.349 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.349 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.349 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: osdmap e38: 6 total, 5 up, 6 in 2024-08-06T17:46:37.349 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.349 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:37 smithi107 ceph-mon[32381]: from='osd.5 ' entity='osd.5' 2024-08-06T17:46:37.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:37.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: osdmap e37: 6 total, 5 up, 6 in 2024-08-06T17:46:37.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: osdmap e38: 6 total, 5 up, 6 in 2024-08-06T17:46:37.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:37 smithi086 ceph-mon[29229]: from='osd.5 ' entity='osd.5' 2024-08-06T17:46:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:46:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: osdmap e37: 6 total, 5 up, 6 in 2024-08-06T17:46:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: osdmap e38: 6 total, 5 up, 6 in 2024-08-06T17:46:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:37 smithi117 ceph-mon[32835]: from='osd.5 ' entity='osd.5' 2024-08-06T17:46:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:46:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:46:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119] boot 2024-08-06T17:46:38.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: osdmap e39: 6 total, 6 up, 6 in 2024-08-06T17:46:38.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:38 smithi086 ceph-mon[29229]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:46:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:46:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119] boot 2024-08-06T17:46:38.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: osdmap e39: 6 total, 6 up, 6 in 2024-08-06T17:46:38.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:38 smithi117 ceph-mon[32835]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: osd.5 [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119] boot 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: osdmap e39: 6 total, 6 up, 6 in 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 5} : dispatch 2024-08-06T17:46:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:38 smithi107 ceph-mon[32381]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-08-06T17:46:39.362 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-08-06T17:46:39.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:39 smithi107 ceph-mon[32381]: osdmap e40: 6 total, 6 up, 6 in 2024-08-06T17:46:39.445 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-08-06T17:46:39.462 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-08-06T17:46:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-08-06T17:46:39.462 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0152657 s, 687 MB/s 2024-08-06T17:46:39.462 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-08-06T17:46:39.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:39 smithi086 ceph-mon[29229]: osdmap e40: 6 total, 6 up, 6 in 2024-08-06T17:46:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:39 smithi117 ceph-mon[32835]: osdmap e40: 6 total, 6 up, 6 in 2024-08-06T17:46:40.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_2 2024-08-06T17:46:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: osdmap e41: 6 total, 6 up, 6 in 2024-08-06T17:46:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:40.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:40.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.530 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: osdmap e41: 6 total, 6 up, 6 in 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:40.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:40.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: osdmap e41: 6 total, 6 up, 6 in 2024-08-06T17:46:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: pgmap v137: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:46:40.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:40.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:40.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:40.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:41.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:41 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi107 2024-08-06T17:46:41.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:41 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi107 to 8543M 2024-08-06T17:46:41.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:41 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi107 2024-08-06T17:46:41.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:41 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi107 to 8543M 2024-08-06T17:46:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:41 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi107 2024-08-06T17:46:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:41 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi107 to 8543M 2024-08-06T17:46:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:42 smithi107 ceph-mon[32381]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:42 smithi086 ceph-mon[29229]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:42 smithi117 ceph-mon[32835]: pgmap v138: 1 pgs: 1 remapped; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:43.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:43 smithi107 ceph-mon[32381]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:43.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:43.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:43.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:43 smithi086 ceph-mon[29229]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:43 smithi117 ceph-mon[32835]: from='client.24236 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:46:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:46:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:46:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:44.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:44 smithi107 ceph-mon[32381]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-06T17:46:44.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:44 smithi086 ceph-mon[29229]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-06T17:46:44.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:44 smithi117 ceph-mon[32835]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 80 KiB/s, 0 objects/s recovering 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/1116903429' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"}]': finished 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: osdmap e42: 7 total, 6 up, 7 in 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:46:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/1848740910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/1116903429' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"}]': finished 2024-08-06T17:46:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: osdmap e42: 7 total, 6 up, 7 in 2024-08-06T17:46:45.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:46:45.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/1848740910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/1116903429' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"} : dispatch 2024-08-06T17:46:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "26ac2f5b-7633-46a8-a433-bab4bef555d7"}]': finished 2024-08-06T17:46:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: osdmap e42: 7 total, 6 up, 7 in 2024-08-06T17:46:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:46:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/1848740910' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:46:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:46 smithi107 ceph-mon[32381]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-06T17:46:46.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:46 smithi086 ceph-mon[29229]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-06T17:46:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:46 smithi117 ceph-mon[32835]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 70 KiB/s, 0 objects/s recovering 2024-08-06T17:46:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:48 smithi086 ceph-mon[29229]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-06T17:46:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:48 smithi117 ceph-mon[32835]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-06T17:46:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:48 smithi107 ceph-mon[32381]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-08-06T17:46:50.785 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:50 smithi107 ceph-mon[32381]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:50.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-06T17:46:50.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:50 smithi117 ceph-mon[32835]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-06T17:46:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:50 smithi086 ceph-mon[29229]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.6"} : dispatch 2024-08-06T17:46:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:51 smithi117 ceph-mon[32835]: Deploying daemon osd.6 on smithi107 2024-08-06T17:46:51.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:51 smithi107 ceph-mon[32381]: Deploying daemon osd.6 on smithi107 2024-08-06T17:46:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:51 smithi086 ceph-mon[29229]: Deploying daemon osd.6 on smithi107 2024-08-06T17:46:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:52 smithi117 ceph-mon[32835]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:52.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:52 smithi107 ceph-mon[32381]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:52 smithi086 ceph-mon[29229]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-08-06T17:46:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:54 smithi117 ceph-mon[32835]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:54 smithi107 ceph-mon[32381]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:55.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:54 smithi086 ceph-mon[29229]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:56.895 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:56 smithi107 ceph-mon[32381]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:56.895 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:56.895 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:56.895 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:57.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:56 smithi086 ceph-mon[29229]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:57.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:57.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:57.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:56 smithi117 ceph-mon[32835]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:58.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:58 smithi107 ceph-mon[32381]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:58.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:58 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:58.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:58 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:58 smithi086 ceph-mon[29229]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:58 smithi117 ceph-mon[32835]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:46:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.344 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 6 on host 'smithi107' 2024-08-06T17:46:59.868 DEBUG:teuthology.orchestra.run.smithi107:osd.6> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service 2024-08-06T17:46:59.871 INFO:tasks.cephadm:Deploying osd.7 on smithi107 with /dev/vg_nvme/lv_1... 2024-08-06T17:46:59.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-08-06T17:46:59.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:46:59.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:46:59.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:46:59.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:46:59.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:46:59.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:46:59.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:46:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:47:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:47:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:00.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:00.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:46:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:47:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]} : dispatch 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:46:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:00.604 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:47:00.922 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:00.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-06T17:47:00.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:00.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: osdmap e43: 7 total, 6 up, 7 in 2024-08-06T17:47:00.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:00.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:00 smithi107 ceph-mon[32381]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:01.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:01.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-06T17:47:01.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:01.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: osdmap e43: 7 total, 6 up, 7 in 2024-08-06T17:47:01.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:01.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:00 smithi086 ceph-mon[29229]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-08-06T17:47:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: from='osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377]' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: osdmap e43: 7 total, 6 up, 7 in 2024-08-06T17:47:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:01.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:00 smithi117 ceph-mon[32835]: from='osd.6 ' entity='osd.6' cmd={"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:01.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-6[48448]: 2024-08-06T17:47:01.068+0000 7fcd9ce40640 -1 osd.6 0 waiting for initial osdmap 2024-08-06T17:47:01.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-6[48448]: 2024-08-06T17:47:01.075+0000 7fcd94c5f640 -1 osd.6 44 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:47:01.915 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: osdmap e44: 7 total, 6 up, 7 in 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: from='osd.6 ' entity='osd.6' 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377] boot 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: osdmap e45: 7 total, 7 up, 7 in 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:01.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:01 smithi107 ceph-mon[32381]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: osdmap e44: 7 total, 6 up, 7 in 2024-08-06T17:47:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: from='osd.6 ' entity='osd.6' 2024-08-06T17:47:02.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377] boot 2024-08-06T17:47:02.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: osdmap e45: 7 total, 7 up, 7 in 2024-08-06T17:47:02.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:01 smithi086 ceph-mon[29229]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: osdmap e44: 7 total, 6 up, 7 in 2024-08-06T17:47:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: from='osd.6 ' entity='osd.6' 2024-08-06T17:47:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: osd.6 [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377] boot 2024-08-06T17:47:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: osdmap e45: 7 total, 7 up, 7 in 2024-08-06T17:47:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 6} : dispatch 2024-08-06T17:47:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:01 smithi117 ceph-mon[32835]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 161 MiB used, 536 GiB / 536 GiB avail 2024-08-06T17:47:02.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:02 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:02.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:02 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:02.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:02.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:02.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:02 smithi107 ceph-mon[32381]: osdmap e46: 7 total, 7 up, 7 in 2024-08-06T17:47:02.926 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-08-06T17:47:03.014 INFO:teuthology.orchestra.run.smithi107.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-08-06T17:47:03.028 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10+0 records in 2024-08-06T17:47:03.028 INFO:teuthology.orchestra.run.smithi107.stderr:10+0 records out 2024-08-06T17:47:03.028 INFO:teuthology.orchestra.run.smithi107.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124871 s, 840 MB/s 2024-08-06T17:47:03.029 INFO:teuthology.orchestra.run.smithi107.stderr:--> Zapping successful for: 2024-08-06T17:47:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:02 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:02 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:02 smithi086 ceph-mon[29229]: osdmap e46: 7 total, 7 up, 7 in 2024-08-06T17:47:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:02 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:02 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:02 smithi117 ceph-mon[32835]: osdmap e46: 7 total, 7 up, 7 in 2024-08-06T17:47:03.768 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:03 smithi107 ceph-mon[32381]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:03 smithi086 ceph-mon[29229]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:03 smithi117 ceph-mon[32835]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:04.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi107:vg_nvme/lv_1 2024-08-06T17:47:04.494 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:47:04.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:04.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: osdmap e47: 7 total, 7 up, 7 in 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi107 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi107 to 5695M 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:04.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: osdmap e47: 7 total, 7 up, 7 in 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi107 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi107 to 5695M 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:05.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: osdmap e47: 7 total, 7 up, 7 in 2024-08-06T17:47:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi107 2024-08-06T17:47:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi107 to 5695M 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:05.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:05 smithi107 ceph-mon[32381]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:05 smithi086 ceph-mon[29229]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:05 smithi117 ceph-mon[32835]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 536 MiB used, 625 GiB / 626 GiB avail 2024-08-06T17:47:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:06 smithi107 ceph-mon[32381]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:06 smithi086 ceph-mon[29229]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:06 smithi117 ceph-mon[32835]: from='client.24263 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi107:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:07.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:07 smithi107 ceph-mon[32381]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:08.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:07 smithi086 ceph-mon[29229]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:07 smithi117 ceph-mon[32835]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/2619691929' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:08 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:08 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"}]': finished 2024-08-06T17:47:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:08 smithi107 ceph-mon[32381]: osdmap e48: 8 total, 7 up, 8 in 2024-08-06T17:47:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:09.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/2619691929' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:09.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:08 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:09.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:08 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"}]': finished 2024-08-06T17:47:09.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:08 smithi086 ceph-mon[29229]: osdmap e48: 8 total, 7 up, 8 in 2024-08-06T17:47:09.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/2619691929' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:08 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"} : dispatch 2024-08-06T17:47:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:08 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1c308454-14da-4e8c-9065-60e3306e30f1"}]': finished 2024-08-06T17:47:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:08 smithi117 ceph-mon[32835]: osdmap e48: 8 total, 7 up, 8 in 2024-08-06T17:47:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/3904387957' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:09 smithi107 ceph-mon[32381]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:10.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/3904387957' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:10.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:09 smithi086 ceph-mon[29229]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/3904387957' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:09 smithi117 ceph-mon[32835]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:12 smithi086 ceph-mon[29229]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:12 smithi117 ceph-mon[32835]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:12.444 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:12 smithi107 ceph-mon[32381]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:13.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:13 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:13.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:13 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:13.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:13 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:13.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:13 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:13 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:13 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:14.303 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:14 smithi107 ceph-mon[32381]: osdmap e49: 8 total, 7 up, 8 in 2024-08-06T17:47:14.303 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:14.303 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:14 smithi107 ceph-mon[32381]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:14 smithi086 ceph-mon[29229]: osdmap e49: 8 total, 7 up, 8 in 2024-08-06T17:47:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:14 smithi086 ceph-mon[29229]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:14 smithi117 ceph-mon[32835]: osdmap e49: 8 total, 7 up, 8 in 2024-08-06T17:47:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:14 smithi117 ceph-mon[32835]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:15.372 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-06T17:47:15.372 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-06T17:47:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.7"} : dispatch 2024-08-06T17:47:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:16.330 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:16 smithi107 ceph-mon[32381]: Deploying daemon osd.7 on smithi107 2024-08-06T17:47:16.331 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:16 smithi107 ceph-mon[32381]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:16.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:16 smithi086 ceph-mon[29229]: Deploying daemon osd.7 on smithi107 2024-08-06T17:47:16.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:16 smithi086 ceph-mon[29229]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:16 smithi117 ceph-mon[32835]: Deploying daemon osd.7 on smithi107 2024-08-06T17:47:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:16 smithi117 ceph-mon[32835]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:17.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:17 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:17.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:17 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:17.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:17 smithi107 ceph-mon[32381]: osdmap e50: 8 total, 7 up, 8 in 2024-08-06T17:47:17.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:17 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:17 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:17 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:17 smithi086 ceph-mon[29229]: osdmap e50: 8 total, 7 up, 8 in 2024-08-06T17:47:17.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:17 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:17 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:17 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:17.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:17 smithi117 ceph-mon[32835]: osdmap e50: 8 total, 7 up, 8 in 2024-08-06T17:47:17.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:17 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:18.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:18 smithi107 ceph-mon[32381]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:18 smithi086 ceph-mon[29229]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:18 smithi117 ceph-mon[32835]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:20 smithi117 ceph-mon[32835]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:20.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:20.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:20 smithi107 ceph-mon[32381]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:20.901 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:20.901 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:20.901 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:20 smithi086 ceph-mon[29229]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:22.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.259 INFO:teuthology.orchestra.run.smithi107.stdout:Created osd(s) 7 on host 'smithi107' 2024-08-06T17:47:23.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:23 smithi107 ceph-mon[32381]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:23.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:23 smithi107 ceph-mon[32381]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:23.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:23.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:23 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.966 DEBUG:teuthology.orchestra.run.smithi107:osd.7> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service 2024-08-06T17:47:23.969 INFO:tasks.cephadm:Deploying osd.8 on smithi117 with /dev/vg_nvme/lv_4... 2024-08-06T17:47:23.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-08-06T17:47:23.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:23 smithi117 ceph-mon[32835]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:23.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:23 smithi117 ceph-mon[32835]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:23.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:23.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:23.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:23 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:23 smithi086 ceph-mon[29229]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:23 smithi086 ceph-mon[29229]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]} : dispatch 2024-08-06T17:47:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:23 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:24.223 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: osdmap e51: 8 total, 7 up, 8 in 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:24.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:24 smithi117 ceph-mon[32835]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: osdmap e51: 8 total, 7 up, 8 in 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:24 smithi107 ceph-mon[32381]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-08-06T17:47:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: osdmap e51: 8 total, 7 up, 8 in 2024-08-06T17:47:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: from='osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213]' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:24 smithi086 ceph-mon[29229]: from='osd.7 ' entity='osd.7' cmd={"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]} : dispatch 2024-08-06T17:47:25.362 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-7[54364]: 2024-08-06T17:47:25.063+0000 7fb865d01640 -1 osd.7 0 waiting for initial osdmap 2024-08-06T17:47:25.362 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-7[54364]: 2024-08-06T17:47:25.070+0000 7fb85db20640 -1 osd.7 52 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: osdmap e52: 8 total, 7 up, 8 in 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: from='osd.7 ' entity='osd.7' 2024-08-06T17:47:25.642 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:25 smithi107 ceph-mon[32381]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:25.806 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: osdmap e52: 8 total, 7 up, 8 in 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: from='osd.7 ' entity='osd.7' 2024-08-06T17:47:25.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:25 smithi117 ceph-mon[32835]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:25.892 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-08-06T17:47:25.905 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-08-06T17:47:25.905 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-08-06T17:47:25.906 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115267 s, 910 MB/s 2024-08-06T17:47:25.906 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi107", "root=default"]}]': finished 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: osdmap e52: 8 total, 7 up, 8 in 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: from='osd.7 ' entity='osd.7' 2024-08-06T17:47:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:25 smithi086 ceph-mon[29229]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-08-06T17:47:26.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-08-06T17:47:26.771 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:26.910 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213] boot 2024-08-06T17:47:26.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: osdmap e53: 8 total, 8 up, 8 in 2024-08-06T17:47:26.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:26 smithi117 ceph-mon[32835]: osdmap e54: 8 total, 8 up, 8 in 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213] boot 2024-08-06T17:47:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: osdmap e53: 8 total, 8 up, 8 in 2024-08-06T17:47:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:26 smithi107 ceph-mon[32381]: osdmap e54: 8 total, 8 up, 8 in 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: osd.7 [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213] boot 2024-08-06T17:47:27.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: osdmap e53: 8 total, 8 up, 8 in 2024-08-06T17:47:27.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 7} : dispatch 2024-08-06T17:47:27.119 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:26 smithi086 ceph-mon[29229]: osdmap e54: 8 total, 8 up, 8 in 2024-08-06T17:47:27.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi107 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi107 to 4271M 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:27.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: osdmap e55: 8 total, 8 up, 8 in 2024-08-06T17:47:27.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:27 smithi117 ceph-mon[32835]: pgmap v175: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi107 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi107 to 4271M 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: osdmap e55: 8 total, 8 up, 8 in 2024-08-06T17:47:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:27 smithi107 ceph-mon[32381]: pgmap v175: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:28.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi107 2024-08-06T17:47:28.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi107 to 4271M 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: osdmap e55: 8 total, 8 up, 8 in 2024-08-06T17:47:28.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:27 smithi086 ceph-mon[29229]: pgmap v175: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:29.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:29 smithi117 ceph-mon[32835]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:29.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:29.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:29.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:29 smithi107 ceph-mon[32381]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:29.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:29 smithi086 ceph-mon[29229]: from='client.24235 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:29.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:29.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:29.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:30 smithi117 ceph-mon[32835]: pgmap v176: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:30.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:30 smithi107 ceph-mon[32381]: pgmap v176: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:30.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:30 smithi086 ceph-mon[29229]: pgmap v176: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/920362457' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"}]': finished 2024-08-06T17:47:31.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: osdmap e56: 9 total, 8 up, 9 in 2024-08-06T17:47:31.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:31.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/3319712983' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/920362457' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"}]': finished 2024-08-06T17:47:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: osdmap e56: 9 total, 8 up, 9 in 2024-08-06T17:47:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:31.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/3319712983' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/920362457' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"} : dispatch 2024-08-06T17:47:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "db0ad585-df38-4d49-a78e-797f0aed4b1d"}]': finished 2024-08-06T17:47:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: osdmap e56: 9 total, 8 up, 9 in 2024-08-06T17:47:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:31.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/3319712983' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:32 smithi107 ceph-mon[32381]: pgmap v178: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:32.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:32 smithi086 ceph-mon[29229]: pgmap v178: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:32.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:32 smithi117 ceph-mon[32835]: pgmap v178: 1 pgs: 1 peering; 0 B data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:34 smithi117 ceph-mon[32835]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:34 smithi107 ceph-mon[32381]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:35.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:34 smithi086 ceph-mon[29229]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:36 smithi107 ceph-mon[32381]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-06T17:47:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:36 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:36.963 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:36 smithi117 ceph-mon[32835]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:36.964 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-06T17:47:36.964 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:36 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:36 smithi086 ceph-mon[29229]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.8"} : dispatch 2024-08-06T17:47:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:36 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:37.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:37 smithi117 ceph-mon[32835]: Deploying daemon osd.8 on smithi117 2024-08-06T17:47:37.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:37 smithi107 ceph-mon[32381]: Deploying daemon osd.8 on smithi117 2024-08-06T17:47:38.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:37 smithi086 ceph-mon[29229]: Deploying daemon osd.8 on smithi117 2024-08-06T17:47:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:38 smithi117 ceph-mon[32835]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:38 smithi107 ceph-mon[32381]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:38 smithi086 ceph-mon[29229]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:40.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:40 smithi117 ceph-mon[32835]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:40 smithi107 ceph-mon[32381]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:40 smithi086 ceph-mon[29229]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:41.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:41.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:41.738 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: osdmap e57: 9 total, 8 up, 9 in 2024-08-06T17:47:41.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:41.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:41.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:41.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: osdmap e57: 9 total, 8 up, 9 in 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: osdmap e57: 9 total, 8 up, 9 in 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:42.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:42 smithi117 ceph-mon[32835]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:42 smithi107 ceph-mon[32381]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:43.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:42 smithi086 ceph-mon[29229]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:44.000 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 8 on host 'smithi117' 2024-08-06T17:47:44.615 DEBUG:teuthology.orchestra.run.smithi117:osd.8> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service 2024-08-06T17:47:44.618 INFO:tasks.cephadm:Deploying osd.9 on smithi117 with /dev/vg_nvme/lv_3... 2024-08-06T17:47:44.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.870 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:44.871 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.871 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.871 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:44.871 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:44 smithi117 ceph-mon[32835]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:44.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:44 smithi107 ceph-mon[32381]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:45.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:47:45.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:45.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:45.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:45.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:44 smithi086 ceph-mon[29229]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]} : dispatch 2024-08-06T17:47:45.265 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:47:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-06T17:47:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: osdmap e58: 9 total, 8 up, 9 in 2024-08-06T17:47:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:46.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:46 smithi086 ceph-mon[29229]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: osdmap e58: 9 total, 8 up, 9 in 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-mon[32835]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: osdmap e58: 9 total, 8 up, 9 in 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: from='osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312]' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: from='osd.8 ' entity='osd.8' cmd={"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:47:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:46 smithi107 ceph-mon[32381]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:46.735 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-8[37670]: 2024-08-06T17:47:46.436+0000 7fe9f76a2640 -1 osd.8 0 waiting for initial osdmap 2024-08-06T17:47:46.735 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:47:46 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-8[37670]: 2024-08-06T17:47:46.445+0000 7fe9ef4c1640 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:47:47.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:47:47.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: osdmap e59: 9 total, 8 up, 9 in 2024-08-06T17:47:47.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: from='osd.8 ' entity='osd.8' 2024-08-06T17:47:47.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: 1.0 scrub starts 2024-08-06T17:47:47.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-mon[32835]: 1.0 scrub ok 2024-08-06T17:47:47.341 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:47:47 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-8[37670]: 2024-08-06T17:47:47.142+0000 7fe9eb0b9640 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: osdmap e59: 9 total, 8 up, 9 in 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: from='osd.8 ' entity='osd.8' 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: 1.0 scrub starts 2024-08-06T17:47:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:47 smithi086 ceph-mon[29229]: 1.0 scrub ok 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: osdmap e59: 9 total, 8 up, 9 in 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: from='osd.8 ' entity='osd.8' 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: 1.0 scrub starts 2024-08-06T17:47:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:47 smithi107 ceph-mon[32381]: 1.0 scrub ok 2024-08-06T17:47:47.597 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-08-06T17:47:47.681 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-08-06T17:47:47.696 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-08-06T17:47:47.696 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-08-06T17:47:47.696 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0128695 s, 815 MB/s 2024-08-06T17:47:47.696 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-08-06T17:47:48.278 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312] boot 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: osdmap e60: 9 total, 9 up, 9 in 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:48 smithi117 ceph-mon[32835]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:48.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-08-06T17:47:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312] boot 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: osdmap e60: 9 total, 9 up, 9 in 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:48 smithi107 ceph-mon[32381]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: osd.8 [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312] boot 2024-08-06T17:47:48.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: osdmap e60: 9 total, 9 up, 9 in 2024-08-06T17:47:48.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 8} : dispatch 2024-08-06T17:47:48.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:48 smithi086 ceph-mon[29229]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 216 MiB used, 715 GiB / 715 GiB avail 2024-08-06T17:47:48.869 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:47:49.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: osdmap e61: 9 total, 9 up, 9 in 2024-08-06T17:47:49.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:49.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:49.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:49.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: osdmap e61: 9 total, 9 up, 9 in 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: osdmap e61: 9 total, 9 up, 9 in 2024-08-06T17:47:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:47:49.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:49.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:49.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:47:49.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:47:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:50 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi117 2024-08-06T17:47:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:50 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi117 to 21184M 2024-08-06T17:47:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:50 smithi107 ceph-mon[32381]: osdmap e62: 9 total, 9 up, 9 in 2024-08-06T17:47:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:50 smithi107 ceph-mon[32381]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:50 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi117 2024-08-06T17:47:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:50 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi117 to 21184M 2024-08-06T17:47:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:50 smithi086 ceph-mon[29229]: osdmap e62: 9 total, 9 up, 9 in 2024-08-06T17:47:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:50 smithi086 ceph-mon[29229]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:50 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi117 2024-08-06T17:47:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:50 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi117 to 21184M 2024-08-06T17:47:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:50 smithi117 ceph-mon[32835]: osdmap e62: 9 total, 9 up, 9 in 2024-08-06T17:47:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:50 smithi117 ceph-mon[32835]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 243 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:51 smithi117 ceph-mon[32835]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:51.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:51 smithi107 ceph-mon[32381]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:51 smithi086 ceph-mon[29229]: from='client.24262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:47:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:47:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:47:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:52.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:52 smithi117 ceph-mon[32835]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:52.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:52 smithi086 ceph-mon[29229]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:52.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:52 smithi107 ceph-mon[32381]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:47:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/89146723' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"}]': finished 2024-08-06T17:47:53.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: osdmap e63: 10 total, 9 up, 10 in 2024-08-06T17:47:53.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:53.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/201960009' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/89146723' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"}]': finished 2024-08-06T17:47:53.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: osdmap e63: 10 total, 9 up, 10 in 2024-08-06T17:47:53.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:53.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/201960009' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/89146723' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"} : dispatch 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b35e8fd4-d68d-4704-bb06-f15928193bdb"}]': finished 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: osdmap e63: 10 total, 9 up, 10 in 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/201960009' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:47:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:54 smithi086 ceph-mon[29229]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-06T17:47:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:54 smithi117 ceph-mon[32835]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-06T17:47:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:54 smithi107 ceph-mon[32381]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 86 KiB/s, 0 objects/s recovering 2024-08-06T17:47:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:47:55.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:47:55.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:55 smithi086 ceph-mon[29229]: osdmap e64: 10 total, 9 up, 10 in 2024-08-06T17:47:55.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:47:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:47:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:55 smithi117 ceph-mon[32835]: osdmap e64: 10 total, 9 up, 10 in 2024-08-06T17:47:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:47:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:47:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:55 smithi107 ceph-mon[32381]: osdmap e64: 10 total, 9 up, 10 in 2024-08-06T17:47:55.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:47:56.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:56 smithi117 ceph-mon[32835]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-08-06T17:47:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:56 smithi086 ceph-mon[29229]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-08-06T17:47:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:56 smithi107 ceph-mon[32381]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-08-06T17:47:58.784 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:58 smithi117 ceph-mon[32835]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:47:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:58 smithi107 ceph-mon[32381]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:47:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:58 smithi086 ceph-mon[29229]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:47:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-06T17:47:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:47:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:47:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-06T17:47:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:47:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.9"} : dispatch 2024-08-06T17:48:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:47:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:00.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:00 smithi117 ceph-mon[32835]: Deploying daemon osd.9 on smithi117 2024-08-06T17:48:00.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:00 smithi117 ceph-mon[32835]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:48:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:00 smithi107 ceph-mon[32381]: Deploying daemon osd.9 on smithi117 2024-08-06T17:48:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:00 smithi107 ceph-mon[32381]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:48:01.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:00 smithi086 ceph-mon[29229]: Deploying daemon osd.9 on smithi117 2024-08-06T17:48:01.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:00 smithi086 ceph-mon[29229]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:48:02.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:02 smithi117 ceph-mon[32835]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:02 smithi107 ceph-mon[32381]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:02 smithi086 ceph-mon[29229]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:03 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:04.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:04.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:04.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:03 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:03 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:04.914 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:04 smithi117 ceph-mon[32835]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:04 smithi107 ceph-mon[32381]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:04 smithi086 ceph-mon[29229]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:05 smithi117 ceph-mon[32835]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:05.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:05 smithi107 ceph-mon[32381]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:05 smithi086 ceph-mon[29229]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:06.907 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 9 on host 'smithi117' 2024-08-06T17:48:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:07.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:07.472 DEBUG:teuthology.orchestra.run.smithi117:osd.9> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service 2024-08-06T17:48:07.475 INFO:tasks.cephadm:Deploying osd.10 on smithi117 with /dev/vg_nvme/lv_2... 2024-08-06T17:48:07.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-08-06T17:48:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:07 smithi117 ceph-mon[32835]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:07 smithi107 ceph-mon[32381]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]} : dispatch 2024-08-06T17:48:08.268 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:07 smithi086 ceph-mon[29229]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:08.485 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:48:09.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:08 smithi117 ceph-mon[32835]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-06T17:48:09.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:08 smithi117 ceph-mon[32835]: osdmap e65: 10 total, 9 up, 10 in 2024-08-06T17:48:09.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:08 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:09.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:08 smithi117 ceph-mon[32835]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:08 smithi117 ceph-mon[32835]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:08 smithi107 ceph-mon[32381]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-06T17:48:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:08 smithi107 ceph-mon[32381]: osdmap e65: 10 total, 9 up, 10 in 2024-08-06T17:48:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:08 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:08 smithi107 ceph-mon[32381]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:08 smithi107 ceph-mon[32381]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:08 smithi086 ceph-mon[29229]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-08-06T17:48:09.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:08 smithi086 ceph-mon[29229]: osdmap e65: 10 total, 9 up, 10 in 2024-08-06T17:48:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:08 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:08 smithi086 ceph-mon[29229]: from='osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576]' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:08 smithi086 ceph-mon[29229]: from='osd.9 ' entity='osd.9' cmd={"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:09.448 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-9[43346]: 2024-08-06T17:48:09.346+0000 7fe6945d6640 -1 osd.9 0 waiting for initial osdmap 2024-08-06T17:48:09.449 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-9[43346]: 2024-08-06T17:48:09.364+0000 7fe68c3f5640 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: osdmap e66: 10 total, 9 up, 10 in 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: from='osd.9 ' entity='osd.9' 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:09 smithi117 ceph-mon[32835]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: osdmap e66: 10 total, 9 up, 10 in 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: from='osd.9 ' entity='osd.9' 2024-08-06T17:48:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:09 smithi107 ceph-mon[32381]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: from='osd.9 ' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: osdmap e66: 10 total, 9 up, 10 in 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:10.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: from='osd.9 ' entity='osd.9' 2024-08-06T17:48:10.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:10.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:09 smithi086 ceph-mon[29229]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 244 MiB used, 804 GiB / 805 GiB avail 2024-08-06T17:48:10.476 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-08-06T17:48:10.561 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-08-06T17:48:10.575 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-08-06T17:48:10.575 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-08-06T17:48:10.576 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123769 s, 847 MB/s 2024-08-06T17:48:10.576 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-08-06T17:48:11.168 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.168 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:10 smithi117 ceph-mon[32835]: osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576] boot 2024-08-06T17:48:11.168 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:10 smithi117 ceph-mon[32835]: osdmap e67: 10 total, 10 up, 10 in 2024-08-06T17:48:11.169 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-08-06T17:48:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:10 smithi107 ceph-mon[32381]: osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576] boot 2024-08-06T17:48:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:10 smithi107 ceph-mon[32381]: osdmap e67: 10 total, 10 up, 10 in 2024-08-06T17:48:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:10 smithi086 ceph-mon[29229]: osd.9 [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576] boot 2024-08-06T17:48:11.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:10 smithi086 ceph-mon[29229]: osdmap e67: 10 total, 10 up, 10 in 2024-08-06T17:48:11.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 9} : dispatch 2024-08-06T17:48:11.681 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:48:12.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: osdmap e68: 10 total, 10 up, 10 in 2024-08-06T17:48:12.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:12.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:12.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: osdmap e68: 10 total, 10 up, 10 in 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:12.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: osdmap e68: 10 total, 10 up, 10 in 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:12.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:12.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:13 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi117 2024-08-06T17:48:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:13 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi117 to 10592M 2024-08-06T17:48:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:13 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi117 2024-08-06T17:48:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:13 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi117 to 10592M 2024-08-06T17:48:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:13 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi117 2024-08-06T17:48:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:13 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi117 to 10592M 2024-08-06T17:48:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:14 smithi117 ceph-mon[32835]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:14 smithi117 ceph-mon[32835]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:14 smithi107 ceph-mon[32381]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:14 smithi107 ceph-mon[32381]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:14 smithi086 ceph-mon[29229]: from='client.24289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:14 smithi086 ceph-mon[29229]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:15 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:15 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:15.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:15 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:15.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:15 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:15 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:15 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: osdmap e69: 10 total, 10 up, 10 in 2024-08-06T17:48:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/984900110' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"}]': finished 2024-08-06T17:48:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: osdmap e70: 11 total, 10 up, 11 in 2024-08-06T17:48:16.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:16.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:16.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/1542938734' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: osdmap e69: 10 total, 10 up, 10 in 2024-08-06T17:48:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/984900110' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"}]': finished 2024-08-06T17:48:16.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: osdmap e70: 11 total, 10 up, 11 in 2024-08-06T17:48:16.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:16.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:16.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/1542938734' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: osdmap e69: 10 total, 10 up, 10 in 2024-08-06T17:48:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/984900110' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"} : dispatch 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c41ae499-4a09-4ead-bcfa-fdf8838f397c"}]': finished 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: osdmap e70: 11 total, 10 up, 11 in 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:16.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/1542938734' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:18.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:18 smithi117 ceph-mon[32835]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:18 smithi107 ceph-mon[32381]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:18 smithi086 ceph-mon[29229]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:20.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:20 smithi117 ceph-mon[32835]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:20 smithi107 ceph-mon[32381]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:20 smithi086 ceph-mon[29229]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:21.768 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:21 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-06T17:48:21.768 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:21 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:21 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-06T17:48:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:21 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:22.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:21 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.10"} : dispatch 2024-08-06T17:48:22.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:21 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: Deploying daemon osd.10 on smithi117 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: osdmap e71: 11 total, 10 up, 11 in 2024-08-06T17:48:22.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:22 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: Deploying daemon osd.10 on smithi117 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: osdmap e71: 11 total, 10 up, 11 in 2024-08-06T17:48:22.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:22 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: Deploying daemon osd.10 on smithi117 2024-08-06T17:48:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:23.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: osdmap e71: 11 total, 10 up, 11 in 2024-08-06T17:48:23.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:22 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:23 smithi117 ceph-mon[32835]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:23 smithi107 ceph-mon[32381]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:23 smithi086 ceph-mon[29229]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:26.891 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:26 smithi117 ceph-mon[32835]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:26.891 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:26.891 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:26.891 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:26 smithi107 ceph-mon[32381]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:26 smithi086 ceph-mon[29229]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:28.722 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: osdmap e72: 11 total, 10 up, 11 in 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:28.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: osdmap e72: 11 total, 10 up, 11 in 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: osdmap e72: 11 total, 10 up, 11 in 2024-08-06T17:48:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.582 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 10 on host 'smithi117' 2024-08-06T17:48:29.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:29.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]} : dispatch 2024-08-06T17:48:30.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:30.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:30.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:30.148 DEBUG:teuthology.orchestra.run.smithi117:osd.10> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service 2024-08-06T17:48:30.151 INFO:tasks.cephadm:Deploying osd.11 on smithi117 with /dev/vg_nvme/lv_1... 2024-08-06T17:48:30.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-08-06T17:48:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-06T17:48:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: osdmap e73: 11 total, 10 up, 11 in 2024-08-06T17:48:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:30.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-mon[32835]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: osdmap e73: 11 total, 10 up, 11 in 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:30 smithi107 ceph-mon[32381]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:31.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:31.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-08-06T17:48:31.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: osdmap e73: 11 total, 10 up, 11 in 2024-08-06T17:48:31.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:31.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: from='osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815]' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:31.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:30 smithi086 ceph-mon[29229]: from='osd.10 ' entity='osd.10' cmd={"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:31.285 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-10[48951]: 2024-08-06T17:48:30.906+0000 7f8642f88640 -1 osd.10 0 waiting for initial osdmap 2024-08-06T17:48:31.285 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:48:30 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-10[48951]: 2024-08-06T17:48:30.920+0000 7f863edaf640 -1 osd.10 74 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:48:31.594 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: osdmap e74: 11 total, 10 up, 11 in 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:31 smithi117 ceph-mon[32835]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: osdmap e74: 11 total, 10 up, 11 in 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:31 smithi107 ceph-mon[32381]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: osdmap e74: 11 total, 10 up, 11 in 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:32.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:31 smithi086 ceph-mon[29229]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 271 MiB used, 894 GiB / 894 GiB avail 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815] boot 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: osdmap e75: 11 total, 11 up, 11 in 2024-08-06T17:48:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:33.241 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815] boot 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: osdmap e75: 11 total, 11 up, 11 in 2024-08-06T17:48:33.242 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:33.332 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-08-06T17:48:33.347 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-08-06T17:48:33.347 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-08-06T17:48:33.347 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128179 s, 818 MB/s 2024-08-06T17:48:33.347 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 2024-08-06T17:48:33.347 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: osd.10 [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815] boot 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: osdmap e75: 11 total, 11 up, 11 in 2024-08-06T17:48:33.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 10} : dispatch 2024-08-06T17:48:34.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-08-06T17:48:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: osdmap e76: 11 total, 11 up, 11 in 2024-08-06T17:48:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.616 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:48:34.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: osdmap e76: 11 total, 11 up, 11 in 2024-08-06T17:48:34.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:34.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:34.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: osdmap e76: 11 total, 11 up, 11 in 2024-08-06T17:48:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:34.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:35 smithi117 ceph-mon[32835]: Detected new or changed devices on smithi117 2024-08-06T17:48:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:35 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi117 to 7061M 2024-08-06T17:48:35.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:35 smithi107 ceph-mon[32381]: Detected new or changed devices on smithi117 2024-08-06T17:48:35.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:35 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi117 to 7061M 2024-08-06T17:48:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:35 smithi086 ceph-mon[29229]: Detected new or changed devices on smithi117 2024-08-06T17:48:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:35 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi117 to 7061M 2024-08-06T17:48:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:36 smithi117 ceph-mon[32835]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:36.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:36 smithi107 ceph-mon[32381]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:36.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:36 smithi086 ceph-mon[29229]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:37 smithi117 ceph-mon[32835]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:37.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:37 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:37 smithi107 ceph-mon[32381]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:37 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:37.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:37 smithi086 ceph-mon[29229]: from='client.24316 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:48:37.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd tree", "states": ["destroyed"], "format": "json"} : dispatch 2024-08-06T17:48:37.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.bootstrap-osd"} : dispatch 2024-08-06T17:48:37.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:37 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:38.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:38 smithi107 ceph-mon[32381]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:38.517 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:38 smithi086 ceph-mon[29229]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:38.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:38 smithi117 ceph-mon[32835]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:39.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/4095381387' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"}]': finished 2024-08-06T17:48:39.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: osdmap e77: 12 total, 11 up, 12 in 2024-08-06T17:48:39.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:39.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/4281221519' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/4095381387' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"}]': finished 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: osdmap e77: 12 total, 11 up, 12 in 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:39.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/4281221519' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/4095381387' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd={"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"} : dispatch 2024-08-06T17:48:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3f543ee4-77a1-48fb-92cc-e920c871a986"}]': finished 2024-08-06T17:48:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: osdmap e77: 12 total, 11 up, 12 in 2024-08-06T17:48:39.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:39.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/4281221519' entity='client.bootstrap-osd' cmd={"prefix": "mon getmap"} : dispatch 2024-08-06T17:48:40.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:40 smithi086 ceph-mon[29229]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:40.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:40 smithi107 ceph-mon[32381]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:40 smithi117 ceph-mon[32835]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:42.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:42 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:42.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:42 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:42.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:42 smithi117 ceph-mon[32835]: osdmap e78: 12 total, 11 up, 12 in 2024-08-06T17:48:42.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:42.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:42 smithi117 ceph-mon[32835]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:42 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:42 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:42 smithi086 ceph-mon[29229]: osdmap e78: 12 total, 11 up, 12 in 2024-08-06T17:48:42.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:42.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:42 smithi086 ceph-mon[29229]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:42 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:42 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:42 smithi107 ceph-mon[32381]: osdmap e78: 12 total, 11 up, 12 in 2024-08-06T17:48:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:42 smithi107 ceph-mon[32381]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:44 smithi117 ceph-mon[32835]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:44 smithi107 ceph-mon[32381]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-06T17:48:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:44 smithi086 ceph-mon[29229]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "osd.11"} : dispatch 2024-08-06T17:48:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:45.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:45 smithi117 ceph-mon[32835]: Deploying daemon osd.11 on smithi117 2024-08-06T17:48:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:45 smithi107 ceph-mon[32381]: Deploying daemon osd.11 on smithi117 2024-08-06T17:48:46.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:45 smithi086 ceph-mon[29229]: Deploying daemon osd.11 on smithi117 2024-08-06T17:48:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:46 smithi107 ceph-mon[32381]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:47.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:46 smithi117 ceph-mon[32835]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:47.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:46 smithi086 ceph-mon[29229]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:48.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:48 smithi117 ceph-mon[32835]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:48 smithi107 ceph-mon[32381]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:49.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:48 smithi086 ceph-mon[29229]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:49.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:49.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:49.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:50 smithi107 ceph-mon[32381]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:51.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:50 smithi117 ceph-mon[32835]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:50 smithi086 ceph-mon[29229]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:52.347 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:52.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:52.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:52.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:52.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:52.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:52.828 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 11 on host 'smithi117' 2024-08-06T17:48:53.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:53 smithi107 ceph-mon[32381]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:53 smithi107 ceph-mon[32381]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:53 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:53.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:53 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:53 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.540 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:53 smithi117 ceph-mon[32835]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.540 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:53 smithi117 ceph-mon[32835]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:53 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:53.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:53 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:53 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.567 DEBUG:teuthology.orchestra.run.smithi117:osd.11> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service 2024-08-06T17:48:53.571 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-08-06T17:48:53.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd stat -f json 2024-08-06T17:48:53.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:53 smithi086 ceph-mon[29229]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:53 smithi086 ceph-mon[29229]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]} : dispatch 2024-08-06T17:48:53.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:53 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:53.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:53 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:53 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:53.832 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: osdmap e79: 12 total, 11 up, 12 in 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:54 smithi086 ceph-mon[29229]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: osdmap e79: 12 total, 11 up, 12 in 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:54 smithi107 ceph-mon[32381]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-08-06T17:48:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: from='osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136]' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: osdmap e79: 12 total, 11 up, 12 in 2024-08-06T17:48:54.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:54.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: from='osd.11 ' entity='osd.11' cmd={"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]} : dispatch 2024-08-06T17:48:54.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-mon[32835]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:54.624 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11[54716]: 2024-08-06T17:48:54.152+0000 7f8b560d4640 -1 osd.11 0 waiting for initial osdmap 2024-08-06T17:48:54.624 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:48:54 smithi117 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11[54716]: 2024-08-06T17:48:54.165+0000 7f8b51efb640 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:48:55.300 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:55.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:55.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:55.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: osdmap e80: 12 total, 11 up, 12 in 2024-08-06T17:48:55.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.301 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:55.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: osdmap e80: 12 total, 11 up, 12 in 2024-08-06T17:48:55.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-08-06T17:48:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:48:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: osdmap e80: 12 total, 11 up, 12 in 2024-08-06T17:48:55.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:55.700 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:48:56.297 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1722966535,"num_in_osds":12,"osd_in_since":1722966518,"num_remapped_pgs":1} 2024-08-06T17:48:56.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:48:56.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136] boot 2024-08-06T17:48:56.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: osdmap e81: 12 total, 12 up, 12 in 2024-08-06T17:48:56.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:56.339 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1207183493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-06T17:48:56.339 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:56 smithi117 ceph-mon[32835]: osdmap e82: 12 total, 12 up, 12 in 2024-08-06T17:48:56.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136] boot 2024-08-06T17:48:56.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: osdmap e81: 12 total, 12 up, 12 in 2024-08-06T17:48:56.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:56.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1207183493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-06T17:48:56.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:56 smithi107 ceph-mon[32381]: osdmap e82: 12 total, 12 up, 12 in 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: osd.11 [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136] boot 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: osdmap e81: 12 total, 12 up, 12 in 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 11} : dispatch 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 298 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1207183493' entity='client.admin' cmd={"prefix": "osd stat", "format": "json"} : dispatch 2024-08-06T17:48:56.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:56 smithi086 ceph-mon[29229]: osdmap e82: 12 total, 12 up, 12 in 2024-08-06T17:48:56.605 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:48:57.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: Adjusting osd_memory_target on smithi117 to 5296M 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.508 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:57 smithi086 ceph-mon[29229]: osdmap e83: 12 total, 12 up, 12 in 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: Adjusting osd_memory_target on smithi117 to 5296M 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:57 smithi107 ceph-mon[32381]: osdmap e83: 12 total, 12 up, 12 in 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"} : dispatch 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: Adjusting osd_memory_target on smithi117 to 5296M 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:48:57.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:57 smithi117 ceph-mon[32835]: osdmap e83: 12 total, 12 up, 12 in 2024-08-06T17:48:58.589 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:48:58.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:58 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:48:58.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:58 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:48:58.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:58 smithi086 ceph-mon[29229]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:48:58.590 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":83,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:48:57.136645+0000","last_up_change":"2024-08-06T17:48:55.152282+0000","last_in_change":"2024-08-06T17:48:38.571167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists","up"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6827","nonce":1298962214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6829","nonce":1298962214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6833","nonce":1298962214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6831","nonce":1298962214}]},"public_addr":"172.21.15.86:6827/1298962214","cluster_addr":"172.21.15.86:6829/1298962214","heartbeat_back_addr":"172.21.15.86:6833/1298962214","heartbeat_front_addr":"172.21.15.86:6831/1298962214","state":["exists","up"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists","up"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists","up"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists","up"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists","up"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists","up"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists","up"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:31.604298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.643288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:47:53.498233+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:21.533915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:41.166300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:57.052249+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:14.930431+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:27.087535+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:47:45.106312+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:07.796649+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.455068+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:53.486493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:48:58.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:58 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:48:58.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:58 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:48:58.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:58 smithi107 ceph-mon[32381]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:48:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:58 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:48:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:58 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:48:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:58 smithi117 ceph-mon[32835]: pgmap v248: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:48:59.220 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-08-06T17:45:05.808831+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 11.760000228881836, 'score_stable': 11.760000228881836, 'optimal_score': 0.17000000178813934, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-08-06T17:48:59.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-08-06T17:48:59.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:48:59.593 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:48:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3649374997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:48:59.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:48:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3649374997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:48:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:48:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3649374997' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:00.516 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:00 smithi086 ceph-mon[29229]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:00.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:00 smithi107 ceph-mon[32381]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:00 smithi117 ceph-mon[32835]: pgmap v249: 1 pgs: 1 peering; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:01.158 INFO:teuthology.orchestra.run.smithi086.stdout:pg_num: 1 2024-08-06T17:49:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4016277031' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-06T17:49:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4016277031' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-06T17:49:01.830 INFO:tasks.cephadm:Setting up client nodes... 2024-08-06T17:49:01.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-06T17:49:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4016277031' entity='client.admin' cmd={"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} : dispatch 2024-08-06T17:49:02.083 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:02.518 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:02 smithi086 ceph-mon[29229]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:02.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:02 smithi107 ceph-mon[32381]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:02.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:02 smithi117 ceph-mon[32835]: pgmap v250: 1 pgs: 1 peering; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:03.986 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-08-06T17:49:03.986 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQAPYrJmO5y8OhAAIpihFHW8YdaHN3zumjo98A== 2024-08-06T17:49:04.547 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-08-06T17:49:04.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-08-06T17:49:04.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-08-06T17:49:04.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-06T17:49:04.852 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:49:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:04 smithi086 ceph-mon[29229]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:49:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:04 smithi107 ceph-mon[32381]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:49:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:04 smithi117 ceph-mon[32835]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s, 0 objects/s recovering 2024-08-06T17:49:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1955776671' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:06.797 INFO:teuthology.orchestra.run.smithi107.stdout:[client.1] 2024-08-06T17:49:06.798 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQASYrJmZYJpLxAAsuamIPItca3IA5joJxYUvQ== 2024-08-06T17:49:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:06 smithi107 ceph-mon[32381]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-06T17:49:06.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:06 smithi086 ceph-mon[29229]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-06T17:49:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:06 smithi117 ceph-mon[32835]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s, 0 objects/s recovering 2024-08-06T17:49:07.390 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2024-08-06T17:49:07.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-08-06T17:49:07.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-08-06T17:49:07.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph auth get-or-create client.2 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-08-06T17:49:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:49:07.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:07 smithi117 ceph-mon[32835]: 1.0 scrub starts 2024-08-06T17:49:07.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:07 smithi117 ceph-mon[32835]: 1.0 scrub ok 2024-08-06T17:49:07.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.107:0/1215024978' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:07 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:07 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:07 smithi086 ceph-mon[29229]: 1.0 scrub starts 2024-08-06T17:49:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:07 smithi086 ceph-mon[29229]: 1.0 scrub ok 2024-08-06T17:49:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.107:0/1215024978' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:07 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:07 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:07 smithi107 ceph-mon[32381]: 1.0 scrub starts 2024-08-06T17:49:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:07 smithi107 ceph-mon[32381]: 1.0 scrub ok 2024-08-06T17:49:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.107:0/1215024978' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:07 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:07 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:08 smithi107 ceph-mon[32381]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-08-06T17:49:08.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:08 smithi117 ceph-mon[32835]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-08-06T17:49:09.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:08 smithi086 ceph-mon[29229]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s, 0 objects/s recovering 2024-08-06T17:49:09.543 INFO:teuthology.orchestra.run.smithi117.stdout:[client.2] 2024-08-06T17:49:09.543 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQAVYrJmgbI/IBAA1aCJhuJiQu59xosoP4xmbg== 2024-08-06T17:49:09.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=24,config=2,health=12,kv=58,logm=4} crc {auth=4060845734,config=2353167246,health=3897446751,kv=124142209,logm=1878855285}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966632745}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876922785}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049621903}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4281334048}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787377295}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875981421}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3051605289}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673095639}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986110193}) 2024-08-06T17:49:09.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379737138}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2855524212}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747672153}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369728602}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909332215}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464633577}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744962034}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=23,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=36} crc {logm=3524352053,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3461133567,mon_config_key=748438212}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2413875663}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1541456438}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=3890553826,monmap=2720365863,osd_metadata=930218092,osd_pg_creating=3839215575,osdmap=4102112106}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3954287620}) 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/3995933348' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:09 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=24,config=2,health=12,kv=58,logm=4} crc {auth=4060845734,config=2353167246,health=3897446751,kv=124142209,logm=1878855285}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966632745}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876922785}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049621903}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4281334048}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787377295}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875981421}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3051605289}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673095639}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986110193}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379737138}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2855524212}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747672153}) 2024-08-06T17:49:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369728602}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909332215}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464633577}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744962034}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=23,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=36} crc {logm=3524352053,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3461133567,mon_config_key=748438212}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2413875663}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1541456438}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=3890553826,monmap=2720365863,osd_metadata=930218092,osd_pg_creating=3839215575,osdmap=4102112106}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3954287620}) 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/3995933348' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:09 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=24,config=2,health=12,kv=58,logm=4} crc {auth=4060845734,config=2353167246,health=3897446751,kv=124142209,logm=1878855285}) 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3966632745}) 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2876922785}) 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049621903}) 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4281334048}) 2024-08-06T17:49:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=787377295}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1875981421}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3051605289}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=673095639}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2986110193}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=379737138}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2855524212}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3747672153}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1369728602}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1909332215}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1464633577}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3744962034}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=23,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=36} crc {logm=3524352053,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3461133567,mon_config_key=748438212}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2413875663}) 2024-08-06T17:49:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1541456438}) 2024-08-06T17:49:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=37,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=45} crc {mon_config_key=3890553826,monmap=2720365863,osd_metadata=930218092,osd_pg_creating=3839215575,osdmap=4102112106}) 2024-08-06T17:49:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3954287620}) 2024-08-06T17:49:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/3995933348' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} : dispatch 2024-08-06T17:49:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:09 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.2", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-08-06T17:49:10.201 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-08-06T17:49:10.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.2.keyring 2024-08-06T17:49:10.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2024-08-06T17:49:10.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph config log 1 --format=json 2024-08-06T17:49:10.493 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:10 smithi086 ceph-mon[29229]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:10 smithi117 ceph-mon[32835]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:10 smithi107 ceph-mon[32381]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:12 smithi086 ceph-mon[29229]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:12.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:49:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:12 smithi107 ceph-mon[32381]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:12 smithi117 ceph-mon[32835]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:13.002 INFO:teuthology.orchestra.run.smithi086.stdout:[{"version":29,"timestamp":"2024-08-06T17:48:56.384270+0000","name":"","changes":[{"name":"osd/host:smithi117/osd_memory_target","previous_value":"7404402141","new_value":"5553301606"}]}] 2024-08-06T17:49:13.003 INFO:tasks.ceph_manager:config epoch is 29 2024-08-06T17:49:13.003 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-08-06T17:49:13.003 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-08-06T17:49:13.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph mgr dump --format=json 2024-08-06T17:49:13.255 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:13.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3885375400' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-06T17:49:13.494 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3885375400' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-06T17:49:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3885375400' entity='client.admin' cmd={"prefix": "config log", "num": 1, "format": "json"} : dispatch 2024-08-06T17:49:14.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:14 smithi086 ceph-mon[29229]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:14 smithi107 ceph-mon[32381]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:14 smithi117 ceph-mon[32835]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s, 0 objects/s recovering 2024-08-06T17:49:15.093 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:49:15.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4216050503' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-06T17:49:15.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4216050503' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-06T17:49:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4216050503' entity='client.admin' cmd={"prefix": "mgr dump", "format": "json"} : dispatch 2024-08-06T17:49:15.723 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":15,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6800","nonce":1182912913},{"type":"v1","addr":"172.21.15.86:6801","nonce":1182912913}]},"active_addr":"172.21.15.86:6801/1182912913","active_change":"2024-08-06T17:43:07.506083+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24110,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.27.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:10.4.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_nginx":{"name":"container_image_nginx","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nginx:1.26.1","min":"","max":"","enum_allowed":[],"desc":"Nginx container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.7.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.51.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"internal_store_backend":{"name":"internal_store_backend","type":"str","level":"dev","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"set internal store backend. for develoment and testing only","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.86:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2505711816}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":3732067317}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":1715545193}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.86:0","nonce":2224003676}]}]} 2024-08-06T17:49:15.728 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-08-06T17:49:15.728 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-06T17:49:15.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:49:15.977 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:16.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:16 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:16.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:16 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:16.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:16 smithi086 ceph-mon[29229]: osdmap e84: 12 total, 12 up, 12 in 2024-08-06T17:49:16.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:16 smithi086 ceph-mon[29229]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:16.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:16 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:16.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:16 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:16.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:16 smithi107 ceph-mon[32381]: osdmap e84: 12 total, 12 up, 12 in 2024-08-06T17:49:16.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:16 smithi107 ceph-mon[32381]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:16 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:16 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:16 smithi117 ceph-mon[32835]: osdmap e84: 12 total, 12 up, 12 in 2024-08-06T17:49:16.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:16 smithi117 ceph-mon[32835]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:17.715 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:49:17.715 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":84,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:49:15.152587+0000","last_up_change":"2024-08-06T17:48:55.152282+0000","last_in_change":"2024-08-06T17:48:38.571167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists","up"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6827","nonce":1298962214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6829","nonce":1298962214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6833","nonce":1298962214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6831","nonce":1298962214}]},"public_addr":"172.21.15.86:6827/1298962214","cluster_addr":"172.21.15.86:6829/1298962214","heartbeat_back_addr":"172.21.15.86:6833/1298962214","heartbeat_front_addr":"172.21.15.86:6831/1298962214","state":["exists","up"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists","up"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists","up"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists","up"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists","up"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists","up"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists","up"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:31.604298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.643288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:47:53.498233+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:21.533915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:41.166300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:57.052249+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:14.930431+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:27.087535+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:15.026001+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:07.796649+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.455068+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:53.486493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:49:18.432 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-08-06T17:49:18.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:49:18.681 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:18.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:18 smithi086 ceph-mon[29229]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:18.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3991668171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:18 smithi107 ceph-mon[32381]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3991668171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:18 smithi117 ceph-mon[32835]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:19.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3991668171' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:20.450 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:49:20.450 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":84,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:49:15.152587+0000","last_up_change":"2024-08-06T17:48:55.152282+0000","last_in_change":"2024-08-06T17:48:38.571167+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":11.760000228881836,"score_stable":11.760000228881836,"optimal_score":0.17000000178813934,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists","up"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6826","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6827","nonce":1298962214}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6828","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6829","nonce":1298962214}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6832","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6833","nonce":1298962214}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6830","nonce":1298962214},{"type":"v1","addr":"172.21.15.86:6831","nonce":1298962214}]},"public_addr":"172.21.15.86:6827/1298962214","cluster_addr":"172.21.15.86:6829/1298962214","heartbeat_back_addr":"172.21.15.86:6833/1298962214","heartbeat_front_addr":"172.21.15.86:6831/1298962214","state":["exists","up"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists","up"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists","up"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists","up"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists","up"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists","up"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists","up"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:31.604298+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.643288+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:47:53.498233+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:21.533915+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:41.166300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:57.052249+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:14.930431+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:27.087535+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:15.026001+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:07.796649+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:30.455068+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:48:53.486493+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:49:20.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:20 smithi086 ceph-mon[29229]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:20.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1668307295' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:20.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:20 smithi117 ceph-mon[32835]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:20.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1668307295' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:20 smithi107 ceph-mon[32381]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1668307295' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:49:21.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-08-06T17:49:21.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-08-06T17:49:21.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-08-06T17:49:21.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-08-06T17:49:21.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-08-06T17:49:21.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-08-06T17:49:21.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-08-06T17:49:21.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-08-06T17:49:21.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-08-06T17:49:21.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-08-06T17:49:21.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-08-06T17:49:21.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-08-06T17:49:21.817 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.923 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.926 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.928 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.928 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.929 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.936 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.942 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.947 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:21.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:22.377 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:22.378 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:22.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:22 smithi086 ceph-mon[29229]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:22 smithi107 ceph-mon[32381]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:22 smithi117 ceph-mon[32835]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:23.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:23.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:23.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: osdmap e85: 12 total, 12 up, 12 in 2024-08-06T17:49:23.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:23.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:23.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:23 smithi086 ceph-mon[29229]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: osdmap e85: 12 total, 12 up, 12 in 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:23 smithi107 ceph-mon[32381]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: osdmap e85: 12 total, 12 up, 12 in 2024-08-06T17:49:24.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:24.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:24.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:23 smithi117 ceph-mon[32835]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:24 smithi086 ceph-mon[29229]: osdmap e86: 12 total, 12 up, 12 in 2024-08-06T17:49:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:24 smithi107 ceph-mon[32381]: osdmap e86: 12 total, 12 up, 12 in 2024-08-06T17:49:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:24 smithi117 ceph-mon[32835]: osdmap e86: 12 total, 12 up, 12 in 2024-08-06T17:49:25.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:25 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:25.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:25 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:25.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:25 smithi086 ceph-mon[29229]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:25 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:25 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:25 smithi107 ceph-mon[32381]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:25 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:25 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:26.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:25 smithi117 ceph-mon[32835]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:26.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:26 smithi086 ceph-mon[29229]: osdmap e87: 12 total, 12 up, 12 in 2024-08-06T17:49:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:26 smithi107 ceph-mon[32381]: osdmap e87: 12 total, 12 up, 12 in 2024-08-06T17:49:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:26 smithi117 ceph-mon[32835]: osdmap e87: 12 total, 12 up, 12 in 2024-08-06T17:49:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:27 smithi107 ceph-mon[32381]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:28.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:27 smithi086 ceph-mon[29229]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:27 smithi117 ceph-mon[32835]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:30.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:30 smithi086 ceph-mon[29229]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:30.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:30 smithi107 ceph-mon[32381]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:30 smithi117 ceph-mon[32835]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:31.298 INFO:teuthology.orchestra.run.smithi086.stdout:137438953514 2024-08-06T17:49:31.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-08-06T17:49:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:32 smithi086 ceph-mon[29229]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:32 smithi107 ceph-mon[32381]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:32.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:32 smithi117 ceph-mon[32835]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:34.139 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T17:49:34.141+0000 7fcc1affd640 0 --2- 172.21.15.86:0/1991175545 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fcc1c104160 0x7fcc1c10c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T17:49:34.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:34 smithi086 ceph-mon[29229]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:34.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:34 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:34.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:34 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:34 smithi107 ceph-mon[32381]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:34 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:34 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:34 smithi117 ceph-mon[32835]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:34 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:34 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:35 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:35 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:35 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:35 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:35 smithi107 ceph-mon[32381]: osdmap e88: 12 total, 12 up, 12 in 2024-08-06T17:49:36.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:35 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:36.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:35 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:36.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:35 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:36.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:35 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:36.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:35 smithi086 ceph-mon[29229]: osdmap e88: 12 total, 12 up, 12 in 2024-08-06T17:49:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:35 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:35 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:35 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:35 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:36.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:35 smithi117 ceph-mon[32835]: osdmap e88: 12 total, 12 up, 12 in 2024-08-06T17:49:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:36 smithi086 ceph-mon[29229]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:36 smithi107 ceph-mon[32381]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:36 smithi117 ceph-mon[32835]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:37.509 INFO:teuthology.orchestra.run.smithi086.stdout:167503724583 2024-08-06T17:49:37.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-08-06T17:49:37.743 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:37 smithi086 ceph-mon[29229]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:37 smithi107 ceph-mon[32381]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:37 smithi117 ceph-mon[32835]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:38 smithi086 ceph-mon[29229]: osdmap e89: 12 total, 12 up, 12 in 2024-08-06T17:49:38.944 INFO:teuthology.orchestra.run.smithi086.stdout:193273528353 2024-08-06T17:49:38.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-08-06T17:49:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:38 smithi107 ceph-mon[32381]: osdmap e89: 12 total, 12 up, 12 in 2024-08-06T17:49:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:38 smithi117 ceph-mon[32835]: osdmap e89: 12 total, 12 up, 12 in 2024-08-06T17:49:39.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:39 smithi086 ceph-mon[29229]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:39 smithi107 ceph-mon[32381]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:39 smithi117 ceph-mon[32835]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:40.258 INFO:teuthology.orchestra.run.smithi086.stdout:90194313269 2024-08-06T17:49:40.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-08-06T17:49:40.983 INFO:teuthology.orchestra.run.smithi086.stdout:257698037785 2024-08-06T17:49:40.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-08-06T17:49:41.798 INFO:teuthology.orchestra.run.smithi086.stdout:111669149744 2024-08-06T17:49:41.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-08-06T17:49:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:42 smithi086 ceph-mon[29229]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:42.902 INFO:teuthology.orchestra.run.smithi086.stdout:34359738430 2024-08-06T17:49:42.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-08-06T17:49:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:42 smithi107 ceph-mon[32381]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:42 smithi117 ceph-mon[32835]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:44.907 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:44 smithi086 ceph-mon[29229]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:44 smithi107 ceph-mon[32381]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:44 smithi117 ceph-mon[32835]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:45.278 INFO:teuthology.orchestra.run.smithi086.stdout:287762808854 2024-08-06T17:49:45.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-08-06T17:49:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:45 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:45 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:45 smithi107 ceph-mon[32381]: osdmap e90: 12 total, 12 up, 12 in 2024-08-06T17:49:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:45 smithi107 ceph-mon[32381]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:46.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:45 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:46.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:45 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:46.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:45 smithi086 ceph-mon[29229]: osdmap e90: 12 total, 12 up, 12 in 2024-08-06T17:49:46.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:45 smithi086 ceph-mon[29229]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:46.043 INFO:teuthology.orchestra.run.smithi086.stdout:55834574907 2024-08-06T17:49:46.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-08-06T17:49:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:45 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:46.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:45 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:46.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:45 smithi117 ceph-mon[32835]: osdmap e90: 12 total, 12 up, 12 in 2024-08-06T17:49:46.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:45 smithi117 ceph-mon[32835]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:46.777 INFO:teuthology.orchestra.run.smithi086.stdout:347892350989 2024-08-06T17:49:46.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-08-06T17:49:47.481 INFO:teuthology.orchestra.run.smithi086.stdout:227633266719 2024-08-06T17:49:47.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-08-06T17:49:47.552 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:47.556 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:47.558 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:47.572 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:47.578 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:48 smithi107 ceph-mon[32381]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:48.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:48 smithi086 ceph-mon[29229]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:48 smithi117 ceph-mon[32835]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:49.589 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:50 smithi086 ceph-mon[29229]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:50.925 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:50 smithi117 ceph-mon[32835]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:50 smithi107 ceph-mon[32381]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:51.764 INFO:teuthology.orchestra.run.smithi086.stdout:322122547217 2024-08-06T17:49:51.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-08-06T17:49:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:52 smithi107 ceph-mon[32381]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:52.505 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:52 smithi086 ceph-mon[29229]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:52.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:52 smithi117 ceph-mon[32835]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:54 smithi086 ceph-mon[29229]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:54 smithi107 ceph-mon[32381]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:54 smithi117 ceph-mon[32835]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:55.205 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:55.206 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:49:55.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:55.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:56.007 INFO:teuthology.orchestra.run.smithi086.stdout:287762808856 2024-08-06T17:49:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:56.215 INFO:teuthology.orchestra.run.smithi086.stdout:167503724587 2024-08-06T17:49:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: osdmap e91: 12 total, 12 up, 12 in 2024-08-06T17:49:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:49:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:49:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4063480335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-06T17:49:56.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2821470763' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-06T17:49:56.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: osdmap e91: 12 total, 12 up, 12 in 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4063480335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2821470763' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-06T17:49:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:49:56.996 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724583 got 167503724587 for osd.5 2024-08-06T17:49:56.996 DEBUG:teuthology.parallel:result is None 2024-08-06T17:49:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: osdmap e91: 12 total, 12 up, 12 in 2024-08-06T17:49:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:49:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:49:57.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4063480335' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 9} : dispatch 2024-08-06T17:49:57.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2821470763' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 5} : dispatch 2024-08-06T17:49:57.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:49:57.852 INFO:teuthology.orchestra.run.smithi086.stdout:193273528357 2024-08-06T17:49:57.852 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:57 smithi086 ceph-mon[29229]: osdmap e92: 12 total, 12 up, 12 in 2024-08-06T17:49:57.852 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:49:57.853 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:49:57.853 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:49:57.853 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:57 smithi086 ceph-mon[29229]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:58.080 INFO:teuthology.orchestra.run.smithi086.stdout:55834574909 2024-08-06T17:49:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:57 smithi117 ceph-mon[32835]: osdmap e92: 12 total, 12 up, 12 in 2024-08-06T17:49:58.143 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:49:58.144 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:49:58.144 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:49:58.144 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:57 smithi117 ceph-mon[32835]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:57 smithi107 ceph-mon[32381]: osdmap e92: 12 total, 12 up, 12 in 2024-08-06T17:49:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:49:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:49:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:49:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:57 smithi107 ceph-mon[32381]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:49:58.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1014550654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-06T17:49:58.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3560726742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-06T17:49:58.877 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574907 got 55834574909 for osd.1 2024-08-06T17:49:58.877 DEBUG:teuthology.parallel:result is None 2024-08-06T17:49:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1014550654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-06T17:49:59.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3560726742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-06T17:49:59.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1014550654' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 6} : dispatch 2024-08-06T17:49:59.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3560726742' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 1} : dispatch 2024-08-06T17:49:59.741 INFO:teuthology.orchestra.run.smithi086.stdout:347892350992 2024-08-06T17:49:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:49:59 smithi086 ceph-mon[29229]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:49:59 smithi117 ceph-mon[32835]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:49:59 smithi107 ceph-mon[32381]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:00.850 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350989 got 347892350992 for osd.11 2024-08-06T17:50:00.850 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:01.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1072753053' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-06T17:50:01.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:00 smithi086 ceph-mon[29229]: overall HEALTH_OK 2024-08-06T17:50:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1072753053' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-06T17:50:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:00 smithi117 ceph-mon[32835]: overall HEALTH_OK 2024-08-06T17:50:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1072753053' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 11} : dispatch 2024-08-06T17:50:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:00 smithi107 ceph-mon[32381]: overall HEALTH_OK 2024-08-06T17:50:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:01 smithi086 ceph-mon[29229]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:01 smithi117 ceph-mon[32835]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:01 smithi107 ceph-mon[32381]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:02.336 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808854 got 287762808856 for osd.9 2024-08-06T17:50:02.337 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:02.546 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:02.722 INFO:teuthology.orchestra.run.smithi086.stdout:90194313275 2024-08-06T17:50:03.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/112043361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-06T17:50:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/112043361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-06T17:50:03.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/112043361' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 2} : dispatch 2024-08-06T17:50:03.699 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:03.730 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528353 got 193273528357 for osd.6 2024-08-06T17:50:03.730 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:04.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:03 smithi086 ceph-mon[29229]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:03 smithi117 ceph-mon[32835]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:04.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:03 smithi107 ceph-mon[32381]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:04.864 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:06.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:06 smithi086 ceph-mon[29229]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:06 smithi107 ceph-mon[32381]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:07.094 INFO:teuthology.orchestra.run.smithi086.stdout:34359738436 2024-08-06T17:50:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:06 smithi117 ceph-mon[32835]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:07.297 INFO:teuthology.orchestra.run.smithi086.stdout:137438953521 2024-08-06T17:50:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3346924789' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-06T17:50:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2339360378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-06T17:50:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3346924789' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-06T17:50:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2339360378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-06T17:50:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3346924789' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 0} : dispatch 2024-08-06T17:50:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2339360378' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 4} : dispatch 2024-08-06T17:50:08.231 INFO:tasks.cephadm.ceph_manager.ceph:need seq 137438953514 got 137438953521 for osd.4 2024-08-06T17:50:08.232 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:08 smithi107 ceph-mon[32381]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:09.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:08 smithi086 ceph-mon[29229]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:08 smithi117 ceph-mon[32835]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:09.561 INFO:teuthology.orchestra.run.smithi086.stdout:111669149750 2024-08-06T17:50:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2901672519' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-06T17:50:09.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2901672519' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-06T17:50:10.009 INFO:tasks.cephadm.ceph_manager.ceph:need seq 90194313269 got 90194313275 for osd.2 2024-08-06T17:50:10.009 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2901672519' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 3} : dispatch 2024-08-06T17:50:10.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:10 smithi086 ceph-mon[29229]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:10 smithi107 ceph-mon[32381]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:10 smithi117 ceph-mon[32835]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:11.825 INFO:teuthology.orchestra.run.smithi086.stdout:227633266724 2024-08-06T17:50:11.863 INFO:teuthology.orchestra.run.smithi086.stdout:322122547221 2024-08-06T17:50:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:12 smithi107 ceph-mon[32381]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2242237613' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-06T17:50:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/828886897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-06T17:50:12.596 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:12 smithi086 ceph-mon[29229]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:12.597 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2242237613' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-06T17:50:12.597 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/828886897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-06T17:50:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:12 smithi117 ceph-mon[32835]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2242237613' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 7} : dispatch 2024-08-06T17:50:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/828886897' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 10} : dispatch 2024-08-06T17:50:12.626 INFO:tasks.cephadm.ceph_manager.ceph:need seq 227633266719 got 227633266724 for osd.7 2024-08-06T17:50:12.627 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:13.485 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738430 got 34359738436 for osd.0 2024-08-06T17:50:13.485 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:13.890 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149744 got 111669149750 for osd.3 2024-08-06T17:50:13.890 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:14.298 INFO:tasks.cephadm.ceph_manager.ceph:need seq 322122547217 got 322122547221 for osd.10 2024-08-06T17:50:14.299 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:14 smithi086 ceph-mon[29229]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:14.935 INFO:teuthology.orchestra.run.smithi086.stdout:257698037793 2024-08-06T17:50:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:14 smithi107 ceph-mon[32381]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:14 smithi117 ceph-mon[32835]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:15.619 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037785 got 257698037793 for osd.8 2024-08-06T17:50:15.619 DEBUG:teuthology.parallel:result is None 2024-08-06T17:50:15.619 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-06T17:50:15.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph pg dump --format=json 2024-08-06T17:50:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3417258925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-06T17:50:15.871 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:15.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3417258925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-06T17:50:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3417258925' entity='client.admin' cmd={"prefix": "osd last-stat-seq", "id": 8} : dispatch 2024-08-06T17:50:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:16 smithi107 ceph-mon[32381]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:17.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:16 smithi086 ceph-mon[29229]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:16 smithi117 ceph-mon[32835]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:17.703 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:50:17.703 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-08-06T17:50:18.433 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":297,"stamp":"2024-08-06T17:50:17.606289+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":333764,"kb_used_data":9488,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124608060,"statfs":{"total":1151940427776,"available":1151598653440,"internally_reserved":0,"allocated":9715712,"data_stored":5556936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002620"},"pg_stats":[{"pgid":"1.0","version":"16'92","reported_seq":128,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-08-06T17:49:56.865092+0000","last_change":"2024-08-06T17:49:06.093357+0000","last_active":"2024-08-06T17:49:56.865092+0000","last_peered":"2024-08-06T17:49:56.865092+0000","last_clean":"2024-08-06T17:49:56.865092+0000","last_became_active":"2024-08-06T17:48:57.147819+0000","last_became_peered":"2024-08-06T17:48:57.147819+0000","last_unstale":"2024-08-06T17:49:56.865092+0000","last_undegraded":"2024-08-06T17:49:56.865092+0000","last_fullsized":"2024-08-06T17:49:56.865092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'92","last_scrub_stamp":"2024-08-06T17:49:06.093264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-06T17:45:06.610863+0000","last_clean_scrub_stamp":"2024-08-06T17:49:06.093264+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-06T17:50:21.422854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716904,"statfs":{"total":95995035648,"available":95966109696,"internally_reserved":0,"allocated":1253376,"data_stored":903833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037793,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716904,"statfs":{"total":95995035648,"available":95966109696,"internally_reserved":0,"allocated":1253376,"data_stored":903853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724591,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149752,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-06T17:50:18.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph pg dump --format=json 2024-08-06T17:50:18.682 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:18.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:18 smithi086 ceph-mon[29229]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:18 smithi107 ceph-mon[32381]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:18 smithi117 ceph-mon[32835]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:19.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:19 smithi086 ceph-mon[29229]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:19 smithi107 ceph-mon[32381]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:20.025 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:19 smithi117 ceph-mon[32835]: from='client.14628 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:20.484 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:50:20.485 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-08-06T17:50:20.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:20 smithi086 ceph-mon[29229]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:20 smithi107 ceph-mon[32381]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:20 smithi117 ceph-mon[32835]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:21.225 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2024-08-06T17:50:19.606662+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":333764,"kb_used_data":9488,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124608060,"statfs":{"total":1151940427776,"available":1151598653440,"internally_reserved":0,"allocated":9715712,"data_stored":5556936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002440"},"pg_stats":[{"pgid":"1.0","version":"16'92","reported_seq":128,"reported_epoch":92,"state":"active+clean","last_fresh":"2024-08-06T17:49:56.865092+0000","last_change":"2024-08-06T17:49:06.093357+0000","last_active":"2024-08-06T17:49:56.865092+0000","last_peered":"2024-08-06T17:49:56.865092+0000","last_clean":"2024-08-06T17:49:56.865092+0000","last_became_active":"2024-08-06T17:48:57.147819+0000","last_became_peered":"2024-08-06T17:48:57.147819+0000","last_unstale":"2024-08-06T17:49:56.865092+0000","last_undegraded":"2024-08-06T17:49:56.865092+0000","last_fullsized":"2024-08-06T17:49:56.865092+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'92","last_scrub_stamp":"2024-08-06T17:49:06.093264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-06T17:45:06.610863+0000","last_clean_scrub_stamp":"2024-08-06T17:49:06.093264+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-06T17:50:21.422854+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350996,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716904,"statfs":{"total":95995035648,"available":95966109696,"internally_reserved":0,"allocated":1253376,"data_stored":903833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547223,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037794,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266726,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28248,"kb_used_data":1224,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716904,"statfs":{"total":95995035648,"available":95966109696,"internally_reserved":0,"allocated":1253376,"data_stored":903853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953523,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149753,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313278,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27724,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717428,"statfs":{"total":95995035648,"available":95966646272,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574914,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738439,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27728,"kb_used_data":704,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717424,"statfs":{"total":95995035648,"available":95966642176,"internally_reserved":0,"allocated":720896,"data_stored":374925,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-06T17:50:21.226 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-06T17:50:21.226 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-08-06T17:50:21.226 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-08-06T17:50:21.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph health --format=json 2024-08-06T17:50:21.481 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:21.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:21 smithi086 ceph-mon[29229]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:21 smithi107 ceph-mon[32381]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:21 smithi117 ceph-mon[32835]: from='client.14634 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:22 smithi107 ceph-mon[32381]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:23.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:22 smithi086 ceph-mon[29229]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:22 smithi117 ceph-mon[32835]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:23.460 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:50:23.460 INFO:teuthology.orchestra.run.smithi086.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-08-06T17:50:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3865126439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-06T17:50:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:23 smithi086 ceph-mon[29229]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3865126439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-06T17:50:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:23 smithi107 ceph-mon[32381]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:24.113 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-08-06T17:50:24.113 INFO:tasks.cephadm:Setup complete, yielding 2024-08-06T17:50:24.113 INFO:teuthology.run_tasks:Running task exec... 2024-08-06T17:50:24.124 INFO:teuthology.task.exec:Executing custom commands... 2024-08-06T17:50:24.124 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi086.front.sepia.ceph.com 2024-08-06T17:50:24.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'while ! ceph balancer status ; do sleep 1 ; done' 2024-08-06T17:50:24.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3865126439' entity='client.admin' cmd={"prefix": "health", "format": "json"} : dispatch 2024-08-06T17:50:24.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:23 smithi117 ceph-mon[32835]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:24.413 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:50:24.413 INFO:teuthology.orchestra.run.smithi086.stdout: "active": true, 2024-08-06T17:50:24.413 INFO:teuthology.orchestra.run.smithi086.stdout: "last_optimize_duration": "0:00:00.000237", 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: "last_optimize_started": "Tue Aug 6 17:50:07 2024", 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: "mode": "upmap", 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: "no_optimization_needed": true, 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: "optimize_result": "Unable to find further optimization, or pool(s) pg_num is decreasing, or distribution is already perfect", 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout: "plans": [] 2024-08-06T17:50:24.414 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:50:24.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer mode crush-compat' 2024-08-06T17:50:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: osdmap e93: 12 total, 12 up, 12 in 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:50:24.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:24.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:24.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:24.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:24.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:24 smithi107 ceph-mon[32381]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: osdmap e93: 12 total, 12 up, 12 in 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:50:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:50:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:25.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:24 smithi086 ceph-mon[29229]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: osdmap e93: 12 total, 12 up, 12 in 2024-08-06T17:50:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:24 smithi117 ceph-mon[32835]: from='client.14646 -' entity='client.admin' cmd=[{"prefix": "balancer status", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:25.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph balancer on' 2024-08-06T17:50:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:25 smithi107 ceph-mon[32381]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush weight-set create-compat", "format": "json"} : dispatch 2024-08-06T17:50:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:25 smithi107 ceph-mon[32381]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:26.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:25 smithi086 ceph-mon[29229]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:26.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush weight-set create-compat", "format": "json"} : dispatch 2024-08-06T17:50:26.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:25 smithi086 ceph-mon[29229]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:26.027 INFO:teuthology.run_tasks:Running task thrashosds... 2024-08-06T17:50:26.038 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-06T17:50:26.038 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms inject socket failures': 2500, 'ms tcp read timeout': 5, 'osd_pool_default_min_size': 2, 'osd_pool_default_size': 2}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'paxos service trim min': 10}, 'osd': {'bluestore zero block detection': True, 'debug ms': 1, 'debug osd': 20, 'osd blocked scrub grace period': 3600, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd map cache size': 1, 'osd max backfills': 6, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd scrub during recovery': False, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(REQUEST_STUCK\\)', '\\(MON_DOWN\\)', '\\(OSD_SLOW_PING_TIME', 'but it is still running', 'objects unfound and apparently lost', 'osd_map_cache_size', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum'], 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']}, 'workunit': {'branch': 'main', 'sha1': '0325fd3c11a225c0fd9eb38fab88aad1370ad779'}} 2024-08-06T17:50:26.038 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 0.25, 'chance_pgpnum_fix': 0.25, 'chance_test_map_discontinuity': 2, 'timeout': 1800, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-08-06T17:50:26.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-08-06T17:50:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:25 smithi117 ceph-mon[32835]: from='client.14652 -' entity='client.admin' cmd=[{"prefix": "balancer mode", "mode": "crush-compat", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:26.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush weight-set create-compat", "format": "json"} : dispatch 2024-08-06T17:50:26.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:25 smithi117 ceph-mon[32835]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:26.287 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: osdmap e94: 12 total, 12 up, 12 in 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush dump", "format": "json"} : dispatch 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: 1.0 scrub starts 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: 1.0 scrub ok 2024-08-06T17:50:26.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:26 smithi107 ceph-mon[32381]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-06T17:50:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: osdmap e94: 12 total, 12 up, 12 in 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush dump", "format": "json"} : dispatch 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: 1.0 scrub starts 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: 1.0 scrub ok 2024-08-06T17:50:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:26 smithi086 ceph-mon[29229]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd='[{"prefix": "osd crush weight-set create-compat", "format": "json"}]': finished 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: osdmap e94: 12 total, 12 up, 12 in 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd crush dump", "format": "json"} : dispatch 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: 1.0 scrub starts 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: 1.0 scrub ok 2024-08-06T17:50:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:26 smithi117 ceph-mon[32835]: from='client.14658 -' entity='client.admin' cmd=[{"prefix": "balancer on", "target": ["mon-mgr", ""]}]: dispatch 2024-08-06T17:50:27.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:27 smithi086 ceph-mon[29229]: osdmap e95: 12 total, 12 up, 12 in 2024-08-06T17:50:27.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:27 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:27.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:27 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:27.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:27 smithi086 ceph-mon[29229]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:27 smithi117 ceph-mon[32835]: osdmap e95: 12 total, 12 up, 12 in 2024-08-06T17:50:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:27 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:27 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:28.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:27 smithi117 ceph-mon[32835]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:27 smithi107 ceph-mon[32381]: osdmap e95: 12 total, 12 up, 12 in 2024-08-06T17:50:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:27 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:27 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:27 smithi107 ceph-mon[32381]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:28.818 INFO:tasks.thrashosds:Beginning thrashosds... 2024-08-06T17:50:28.818 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-08-06T17:50:28.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph pg dump --format=json 2024-08-06T17:50:29.080 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: osdmap e96: 12 total, 12 up, 12 in 2024-08-06T17:50:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' 2024-08-06T17:50:29.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:29.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:28 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: osdmap e96: 12 total, 12 up, 12 in 2024-08-06T17:50:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' 2024-08-06T17:50:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:29.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:29.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:28 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:29.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: osdmap e96: 12 total, 12 up, 12 in 2024-08-06T17:50:29.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' 2024-08-06T17:50:29.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:50:29.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:50:29.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:50:29.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:28 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:50:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:29 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:29 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:29 smithi086 ceph-mon[29229]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:29 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:29 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:29 smithi117 ceph-mon[32835]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:29 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:29 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:29 smithi107 ceph-mon[32381]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:30.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:30 smithi086 ceph-mon[29229]: osdmap e97: 12 total, 12 up, 12 in 2024-08-06T17:50:31.086 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:50:31.087 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-08-06T17:50:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:30 smithi117 ceph-mon[32835]: osdmap e97: 12 total, 12 up, 12 in 2024-08-06T17:50:31.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:30 smithi107 ceph-mon[32381]: osdmap e97: 12 total, 12 up, 12 in 2024-08-06T17:50:31.769 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-08-06T17:50:29.729513+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":333976,"kb_used_data":9404,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124607848,"statfs":{"total":1151940427776,"available":1151598436352,"internally_reserved":0,"allocated":9629696,"data_stored":5568679,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.121619"},"pg_stats":[{"pgid":"1.0","version":"16'92","reported_seq":138,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-08-06T17:50:25.850473+0000","last_change":"2024-08-06T17:50:25.850528+0000","last_active":"2024-08-06T17:50:25.850473+0000","last_peered":"2024-08-06T17:50:25.850473+0000","last_clean":"2024-08-06T17:50:25.850473+0000","last_became_active":"2024-08-06T17:48:57.147819+0000","last_became_peered":"2024-08-06T17:48:57.147819+0000","last_unstale":"2024-08-06T17:50:25.850473+0000","last_undegraded":"2024-08-06T17:50:25.850473+0000","last_fullsized":"2024-08-06T17:50:25.850473+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'92","last_scrub_stamp":"2024-08-06T17:50:25.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-06T17:45:06.610863+0000","last_clean_scrub_stamp":"2024-08-06T17:50:25.850383+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-06T17:51:48.185195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350998,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":911210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547225,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":382302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808863,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":382302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037796,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":647168,"data_stored":350673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266728,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1236,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1265664,"data_stored":911230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528364,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":382302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724594,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27760,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717392,"statfs":{"total":95995035648,"available":95966609408,"internally_reserved":0,"allocated":647168,"data_stored":350673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953525,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":382302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":647168,"data_stored":350673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313280,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27736,"kb_used_data":716,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717416,"statfs":{"total":95995035648,"available":95966633984,"internally_reserved":0,"allocated":733184,"data_stored":382302,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574916,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":391355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27748,"kb_used_data":728,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717404,"statfs":{"total":95995035648,"available":95966621696,"internally_reserved":0,"allocated":745472,"data_stored":391355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-06T17:50:31.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph pg dump --format=json 2024-08-06T17:50:32.022 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:32.460 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:32 smithi086 ceph-mon[29229]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:32.460 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:32 smithi086 ceph-mon[29229]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:32.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:32 smithi107 ceph-mon[32381]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:32.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:32 smithi107 ceph-mon[32381]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:32.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:32 smithi117 ceph-mon[32835]: from='client.14670 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:32.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:32 smithi117 ceph-mon[32835]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:33.819 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:50:33.819 INFO:teuthology.orchestra.run.smithi086.stderr:dumped all 2024-08-06T17:50:34.538 INFO:teuthology.orchestra.run.smithi086.stdout:{"pg_ready":true,"pg_map":{"version":310,"stamp":"2024-08-06T17:50:33.609487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":2,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":2,"kb":1124941824,"kb_used":333576,"kb_used_data":8852,"kb_used_omap":18,"kb_used_meta":324077,"kb_avail":1124608248,"statfs":{"total":1151940427776,"available":1151598845952,"internally_reserved":0,"allocated":9064448,"data_stored":5411015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19141,"internal_metadata":331855163},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.888439"},"pg_stats":[{"pgid":"1.0","version":"16'92","reported_seq":146,"reported_epoch":97,"state":"active+clean","last_fresh":"2024-08-06T17:50:30.452297+0000","last_change":"2024-08-06T17:50:25.850528+0000","last_active":"2024-08-06T17:50:30.452297+0000","last_peered":"2024-08-06T17:50:30.452297+0000","last_clean":"2024-08-06T17:50:30.452297+0000","last_became_active":"2024-08-06T17:48:57.147819+0000","last_became_peered":"2024-08-06T17:48:57.147819+0000","last_unstale":"2024-08-06T17:50:30.452297+0000","last_undegraded":"2024-08-06T17:50:30.452297+0000","last_fullsized":"2024-08-06T17:50:30.452297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":14,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"16'92","last_scrub_stamp":"2024-08-06T17:50:25.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-08-06T17:45:06.610863+0000","last_clean_scrub_stamp":"2024-08-06T17:50:25.850383+0000","objects_scrubbed":2,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-08-06T17:51:48.185195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11],"acting":[7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1586784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3727360,"data_stored":3702496,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":2,"acting":2,"num_store_stats":7}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350999,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28184,"kb_used_data":1164,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716968,"statfs":{"total":95995035648,"available":95966175232,"internally_reserved":0,"allocated":1191936,"data_stored":887218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":75,"seq":322122547226,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27792,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717360,"statfs":{"total":95995035648,"available":95966576640,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[1,2,3,4,5,6,7,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808864,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037797,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":53,"seq":227633266729,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28196,"kb_used_data":1176,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716956,"statfs":{"total":95995035648,"available":95966162944,"internally_reserved":0,"allocated":1204224,"data_stored":894875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":45,"seq":193273528365,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724595,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":32,"seq":137438953526,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27776,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717376,"statfs":{"total":95995035648,"available":95966593024,"internally_reserved":0,"allocated":659456,"data_stored":358310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149755,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27764,"kb_used_data":632,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717388,"statfs":{"total":95995035648,"available":95966605312,"internally_reserved":0,"allocated":647168,"data_stored":350673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":27654600},"hb_peers":[1,2,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":21,"seq":90194313281,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27780,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717372,"statfs":{"total":95995035648,"available":95966588928,"internally_reserved":0,"allocated":659456,"data_stored":358310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":27680,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717472,"statfs":{"total":95995035648,"available":95966691328,"internally_reserved":0,"allocated":671744,"data_stored":365947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":528928,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-08-06T17:50:34.538 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-08-06T17:50:34.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump 2024-08-06T17:50:34.793 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:34 smithi086 ceph-mon[29229]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:34 smithi107 ceph-mon[32381]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:34 smithi117 ceph-mon[32835]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:35.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:35 smithi086 ceph-mon[29229]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:35 smithi107 ceph-mon[32381]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:35 smithi117 ceph-mon[32835]: from='client.14676 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 97 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:fsid fd7e697e-541a-11ef-bccc-c7b262605968 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:created 2024-08-06T17:42:16.357349+0000 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:modified 2024-08-06T17:50:29.724747+0000 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-08-06T17:50:36.542 INFO:teuthology.orchestra.run.smithi086.stdout:crush_version 27 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:full_ratio 0.95 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:backfillfull_ratio 0.9 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:nearfull_ratio 0.85 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:require_min_compat_client luminous 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:min_compat_client jewel 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:require_osd_release squid 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:stretch_mode_enabled false 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:pool 1 '.mgr' replicated size 2 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 17 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 11.76 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:max_osd 12 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730] [v2:172.21.15.86:6804/1557268730,v1:172.21.15.86:6805/1557268730] exists,up 2ab66bcc-745f-4928-8cff-7095957437c5 2024-08-06T17:50:36.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100] [v2:172.21.15.86:6812/1859701100,v1:172.21.15.86:6813/1859701100] exists,up 331fb57c-4513-4398-afda-0ba9442c0a06 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493] [v2:172.21.15.86:6820/1080990493,v1:172.21.15.86:6821/1080990493] exists,up 51280800-811c-42ce-97c0-aa3787f5a681 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214] [v2:172.21.15.86:6828/1298962214,v1:172.21.15.86:6829/1298962214] exists,up 121c9b13-fa41-4680-8424-c96f1ddcec09 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981] [v2:172.21.15.107:6802/1874359981,v1:172.21.15.107:6803/1874359981] exists,up a1dd2693-1793-4b19-84da-e020a8252b4d 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119] [v2:172.21.15.107:6810/1512731119,v1:172.21.15.107:6811/1512731119] exists,up 27dd75cf-526a-44b5-8f5e-c53573212054 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377] [v2:172.21.15.107:6818/949622377,v1:172.21.15.107:6819/949622377] exists,up 26ac2f5b-7633-46a8-a433-bab4bef555d7 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.7 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213] [v2:172.21.15.107:6826/1970096213,v1:172.21.15.107:6827/1970096213] exists,up 1c308454-14da-4e8c-9065-60e3306e30f1 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312] [v2:172.21.15.117:6802/1266762312,v1:172.21.15.117:6803/1266762312] exists,up db0ad585-df38-4d49-a78e-797f0aed4b1d 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.9 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576] [v2:172.21.15.117:6810/170095576,v1:172.21.15.117:6811/170095576] exists,up b35e8fd4-d68d-4704-bb06-f15928193bdb 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.10 up in weight 1 up_from 75 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815] [v2:172.21.15.117:6818/678280815,v1:172.21.15.117:6819/678280815] exists,up c41ae499-4a09-4ead-bcfa-fdf8838f397c 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:osd.11 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136] [v2:172.21.15.117:6826/3154565136,v1:172.21.15.117:6827/3154565136] exists,up 3f543ee4-77a1-48fb-92cc-e920c871a986 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/2391299811 expires 2024-08-07T17:43:07.505980+0000 2024-08-06T17:50:36.544 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/2391299811 expires 2024-08-07T17:43:07.505980+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1285673635 expires 2024-08-07T17:43:07.505980+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/3531235780 expires 2024-08-07T17:42:44.380746+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2979281859 expires 2024-08-07T17:42:44.380746+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/1213671535 expires 2024-08-07T17:43:07.505980+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6801/3936694336 expires 2024-08-07T17:42:44.380746+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:6800/3936694336 expires 2024-08-07T17:42:44.380746+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/2397207804 expires 2024-08-07T17:43:07.505980+0000 2024-08-06T17:50:36.545 INFO:teuthology.orchestra.run.smithi086.stdout:blocklist 172.21.15.86:0/4243608842 expires 2024-08-07T17:42:44.380746+0000 2024-08-06T17:50:36.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:36 smithi086 ceph-mon[29229]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:36.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2046350779' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-06T17:50:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:36 smithi107 ceph-mon[32381]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2046350779' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-06T17:50:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:36 smithi117 ceph-mon[32835]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2046350779' entity='client.admin' cmd={"prefix": "osd dump"} : dispatch 2024-08-06T17:50:37.164 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 53 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6802/1557268730,v1:172.21.15.86:6803/1557268730] [v2:172.21.15.86:6804/1557268730,v1:172.21.15.86:6805/1557268730] exists,up 2ab66bcc-745f-4928-8cff-7095957437c5', 'osd.1 up in weight 1 up_from 13 up_thru 15 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6810/1859701100,v1:172.21.15.86:6811/1859701100] [v2:172.21.15.86:6812/1859701100,v1:172.21.15.86:6813/1859701100] exists,up 331fb57c-4513-4398-afda-0ba9442c0a06', 'osd.2 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6818/1080990493,v1:172.21.15.86:6819/1080990493] [v2:172.21.15.86:6820/1080990493,v1:172.21.15.86:6821/1080990493] exists,up 51280800-811c-42ce-97c0-aa3787f5a681', 'osd.3 up in weight 1 up_from 26 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.86:6826/1298962214,v1:172.21.15.86:6827/1298962214] [v2:172.21.15.86:6828/1298962214,v1:172.21.15.86:6829/1298962214] exists,up 121c9b13-fa41-4680-8424-c96f1ddcec09', 'osd.4 up in weight 1 up_from 32 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6800/1874359981,v1:172.21.15.107:6801/1874359981] [v2:172.21.15.107:6802/1874359981,v1:172.21.15.107:6803/1874359981] exists,up a1dd2693-1793-4b19-84da-e020a8252b4d', 'osd.5 up in weight 1 up_from 39 up_thru 40 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6808/1512731119,v1:172.21.15.107:6809/1512731119] [v2:172.21.15.107:6810/1512731119,v1:172.21.15.107:6811/1512731119] exists,up 27dd75cf-526a-44b5-8f5e-c53573212054', 'osd.6 up in weight 1 up_from 45 up_thru 46 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6816/949622377,v1:172.21.15.107:6817/949622377] [v2:172.21.15.107:6818/949622377,v1:172.21.15.107:6819/949622377] exists,up 26ac2f5b-7633-46a8-a433-bab4bef555d7', 'osd.7 up in weight 1 up_from 53 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.107:6824/1970096213,v1:172.21.15.107:6825/1970096213] [v2:172.21.15.107:6826/1970096213,v1:172.21.15.107:6827/1970096213] exists,up 1c308454-14da-4e8c-9065-60e3306e30f1', 'osd.8 up in weight 1 up_from 60 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/1266762312,v1:172.21.15.117:6801/1266762312] [v2:172.21.15.117:6802/1266762312,v1:172.21.15.117:6803/1266762312] exists,up db0ad585-df38-4d49-a78e-797f0aed4b1d', 'osd.9 up in weight 1 up_from 67 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/170095576,v1:172.21.15.117:6809/170095576] [v2:172.21.15.117:6810/170095576,v1:172.21.15.117:6811/170095576] exists,up b35e8fd4-d68d-4704-bb06-f15928193bdb', 'osd.10 up in weight 1 up_from 75 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/678280815,v1:172.21.15.117:6817/678280815] [v2:172.21.15.117:6818/678280815,v1:172.21.15.117:6819/678280815] exists,up c41ae499-4a09-4ead-bcfa-fdf8838f397c', 'osd.11 up in weight 1 up_from 81 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/3154565136,v1:172.21.15.117:6825/3154565136] [v2:172.21.15.117:6826/3154565136,v1:172.21.15.117:6827/3154565136] exists,up 3f543ee4-77a1-48fb-92cc-e920c871a986'] 2024-08-06T17:50:37.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon mon.a config show 2024-08-06T17:50:37.412 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:37.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:37 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:37.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:37 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:37 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:37 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:37 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:37 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:38.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:38 smithi086 ceph-mon[29229]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:38.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:38 smithi086 ceph-mon[29229]: osdmap e98: 12 total, 12 up, 12 in 2024-08-06T17:50:38.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:38 smithi107 ceph-mon[32381]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:38.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:38 smithi107 ceph-mon[32381]: osdmap e98: 12 total, 12 up, 12 in 2024-08-06T17:50:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:38 smithi117 ceph-mon[32835]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:39.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:38 smithi117 ceph-mon[32835]: osdmap e98: 12 total, 12 up, 12 in 2024-08-06T17:50:39.166 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:50:39.166 INFO:teuthology.orchestra.run.smithi086.stdout: "name": "mon.a", 2024-08-06T17:50:39.166 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster": "ceph", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "admin_socket_mode": "", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_allow_insecure_global_id_reclaim": "true", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_client_required": "cephx, none", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_cluster_required": "cephx", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_debug": "true", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_service_required": "cephx", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_service_ticket_ttl": "240.000000", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "auth_supported": "", 2024-08-06T17:50:39.167 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio": "true", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_max_queue_depth": "1024", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_poll_ms": "250", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_aio_reap_max": "16", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_async_discard_threads": "0", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_block_size": "4096", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio": "false", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_debug_inflight_ios": "false", 2024-08-06T17:50:39.168 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_enable_discard": "false", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_flock_retry": "3", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_flock_retry_interval": "0.100000", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_inject_crash": "0", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_inject_crash_flush_delay": "2", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring": "false", 2024-08-06T17:50:39.169 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring_hipri": "false", 2024-08-06T17:50:39.170 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_ioring_sqthread_poll": "false", 2024-08-06T17:50:39.170 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_read_buffer_alignment": "4096", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bdev_type": "", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_alloc_size": "1048576", 2024-08-06T17:50:39.171 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_allocator": "hybrid", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_buffered_io": "true", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_for_zeros": "false", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_volume_selector_often": "false", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_compact_log_sync": "false", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_compact_min_size": "16777216", 2024-08-06T17:50:39.172 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_log_replay_check_allocations": "true", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_max_log_runway": "4194304", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_max_prefetch": "1048576", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_min_flush_size": "524288", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_min_log_runway": "1048576", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_replay_recovery": "false", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_shared_alloc_size": "65536", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluefs_sync_write": "false", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-08-06T17:50:39.173 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_allocation_from_file": "true", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_allocator": "hybrid", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_async_db_compaction": "true", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_blobid_prealloc": "10240", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_create": "true", 2024-08-06T17:50:39.174 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_create": "false", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_path": "", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_db_size": "0", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_path": "", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_preallocate_file": "false", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_size": "107374182400", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_create": "false", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_path": "", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_block_wal_size": "100663296", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs": "true", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-08-06T17:50:39.175 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_env_mirror": "false", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_autotune": "true", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size": "0", 2024-08-06T17:50:39.176 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cache_type": "2q", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_clone_cow": "true", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_algorithm": "snappy", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size": "0", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size": "0", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-08-06T17:50:39.177 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_mode": "none", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_csum_type": "crc32c", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_enforce_min_alloc_size": "0", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_enforce_settings": "default", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_freelist": "false", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_fsck_abort": "false", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_inject_read_err": "false", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_legacy_omap": "false", 2024-08-06T17:50:39.178 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_prefill": "0.000000", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_small_allocations": "0", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_default_buffered_read": "true", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_default_buffered_write": "false", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops": "0", 2024-08-06T17:50:39.179 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_elastic_shared_blobs": "true", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fail_eio": "false", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mkfs": "true", 2024-08-06T17:50:39.180 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mount": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_umount": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_ignore_data_csum": "false", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-08-06T17:50:39.181 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_kvbackend": "rocksdb", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_log_op_age": "5.000000", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_alloc_size": "0", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size": "0", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_defer_interval": "3.000000", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_max_deferred_txc": "32", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size": "0", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-08-06T17:50:39.182 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_nid_prealloc": "1024", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size": "0", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_qfsck_on_mount": "true", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_retry_disk_reads": "3", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_cf": "true", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_rocksdb_options_annex": "", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_coremask": "0x1", 2024-08-06T17:50:39.183 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_io_sleep": "5", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_max_io_completion": "0", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_spdk_mem": "512", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_sync_submit_transaction": "false", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_bytes": "67108864", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io": "0", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_tracing": "false", 2024-08-06T17:50:39.184 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_reserved": "0", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "bluestore_zero_block_detection": "false", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_action_update_interval": "2", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-08-06T17:50:39.185 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_mount_timeout": "10", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_cluster_require_signatures": "false", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_cluster_require_version": "2", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_require_signatures": "false", 2024-08-06T17:50:39.186 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_require_version": "2", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_service_require_signatures": "false", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_service_require_version": "2", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "cephx_sign_messages": "true", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "chdir": "", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_acl_type": "", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_asio_thread_count": "2", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_cache_mid": "0.750000", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_cache_size": "16384", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_caps_release_delay": "5", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_check_pool_perm": "true", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_collect_and_send_global_metrics": "false", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_force_sync_read": "false", 2024-08-06T17:50:39.187 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_getattr_caps": "false", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_inject_features": "", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_debug_inject_tick_delay": "0", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_die_on_failed_remount": "false", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_dirsize_rbytes": "true", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_force_lazyio": "false", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_fs": "", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_inject_fixed_oldest_tid": "false", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_inject_release_failure": "false", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_max_inline_size": "4096", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_max_retries_on_remount_failure": "5", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mds_namespace": "", 2024-08-06T17:50:39.188 INFO:teuthology.orchestra.run.smithi086.stdout: "client_metadata": "", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_gid": "-1", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_timeout": "300", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mount_uid": "-1", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_mountpoint": "/", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_notify_timeout": "10", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc": "true", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_dirty": "104857600", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_dirty_age": "5.000000", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_max_objects": "1000", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_size": "209715200", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_oc_target_dirty": "8388608", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_permissions": "true", 2024-08-06T17:50:39.189 INFO:teuthology.orchestra.run.smithi086.stdout: "client_quota": "true", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_quota_df": "true", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_max_bytes": "0", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_max_periods": "4", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_readahead_min": "131072", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_reconnect_stale": "false", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_shutdown_timeout": "30", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_snapdir": ".snap", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_tick_interval": "1", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_trace": "", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_try_dentry_invalidate": "false", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_use_faked_inos": "false", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "client_use_random_mds": "false", 2024-08-06T17:50:39.190 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog": "false", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_graylog_port": "12201", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_monitors": "default=true", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog": "false", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "clog_to_syslog_level": "info", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_addr": "-", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_network": "", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "cluster_network_interface": "", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_isal": "false", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_level": "5", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zlib_winsize": "-15", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "compressor_zstd_level": "1", 2024-08-06T17:50:39.191 INFO:teuthology.orchestra.run.smithi086.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_alien_op_num_threads": "6", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_alien_thread_cpu_cores": "", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_osd_obc_lru_size": "512", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_osd_stat_interval": "0", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_seastar_cpu_cores": "", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crimson_seastar_num_threads": "0", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location": "", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location_hook": "", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "crush_location_hook_timeout": "10", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "daemonize": "false", 2024-08-06T17:50:39.192 INFO:teuthology.orchestra.run.smithi086.stdout: "daos_pool": "tank", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "dbstore_db_name_prefix": "dbstore", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_alienstore": "0/5", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_allow_any_pool_priority": "false", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asok": "1/5", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asok_assert_abort": "false", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_asserts_on_shutdown": "true", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_auth": "1/5", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bdev": "1/3", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bluefs": "1/5", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_bluestore": "1/5", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_buffer": "0/1", 2024-08-06T17:50:39.193 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_ceph_dedup": "0/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_ceph_exporter": "1/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cephfs_mirror": "0/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cephsqlite": "0/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_client": "0/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_compressor": "1/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_context": "0/1", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_crimson_interrupt": "0/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_crush": "1/1", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_crypto": "1/5", 2024-08-06T17:50:39.194 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_cyanstore": "0/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_deliberately_leak_memory": "false", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_disable_randomized_ping": "false", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_dpdk": "1/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_eventtrace": "1/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_filer": "0/1", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_filestore": "1/3", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_finisher": "1/1", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_fuse": "1/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_heartbeat_testing_span": "0", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_heartbeatmap": "1/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_immutable_obj_cache": "0/5", 2024-08-06T17:50:39.195 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_javaclient": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_journal": "1/3", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_journaler": "0/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_kstore": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_lockdep": "0/1", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mclock": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_balancer": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_locker": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_log": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_log_expire": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_migrator": "1/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mds_quiesce": "3/5", 2024-08-06T17:50:39.196 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_memstore": "1/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mgr": "2/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mgrc": "1/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_mon": "20/20", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_monc": "0/10", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_ms": "1/1", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_none": "0/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objclass": "0/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objectcacher": "0/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_objecter": "0/1", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_optracker": "0/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_osd": "1/5", 2024-08-06T17:50:39.197 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_paxos": "20/20", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_perfcounter": "1/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_prioritycache": "1/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rados": "0/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd": "0/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_mirror": "0/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_pwl": "0/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rbd_replay": "0/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_refs": "0/0", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_reserver": "1/1", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw": "1/5", 2024-08-06T17:50:39.198 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_access": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_datacache": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_dbstore": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_flight": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_lifecycle": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_notification": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rgw_sync": "1/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_rocksdb": "4/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore": "0/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_backref": "0/5", 2024-08-06T17:50:39.199 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_cache": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_cleaner": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_device": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_epm": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_journal": "0/5", 2024-08-06T17:50:39.201 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_lba": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_odata": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_omap": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_onode": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_t": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_seastore_tm": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_striper": "0/1", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_test": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_throttle": "1/1", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_timer": "0/1", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_tp": "0/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "debug_trace": "1/5", 2024-08-06T17:50:39.202 INFO:teuthology.orchestra.run.smithi086.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "device_failure_prediction_mode": "none", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_graylog": "false", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_journald": "false", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_stderr": "true", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "err_to_syslog": "false", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "event_tracing": "false", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_addr": "0.0.0.0", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_cert_file": "", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_http_port": "9926", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_key_file": "", 2024-08-06T17:50:39.203 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_prio_limit": "5", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_sort_metrics": "true", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "exporter_stats_period": "5", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "fake_statfs_for_testing": "0", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "fatal_signal_handlers": "true", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filer_max_purge_ops": "10", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filer_max_truncate_ops": "128", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_apply_finisher_threads": "1", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_blackhole": "false", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_btrfs_clone_range": "true", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_btrfs_snap": "true", 2024-08-06T17:50:39.204 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_caller_concurrency": "10", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_collect_device_partition_information": "true", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_commit_timeout": "600.000000", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_inject_read_err": "false", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_omap_check": "false", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_debug_verify_split": "false", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_dump_file": "", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fadvise": "true", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fail_eio": "true", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fd_cache_shards": "16", 2024-08-06T17:50:39.205 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fd_cache_size": "128", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fiemap": "false", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fiemap_threshold": "4096", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_index_retry_probability": "0.000000", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_inject_stall": "0", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_parallel": "false", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_trailing": "false", 2024-08-06T17:50:39.206 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_journal_writeahead": "false", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_kill_at": "0", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size": "0", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs": "0", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_other": "2", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_sync_interval": "5.000000", 2024-08-06T17:50:39.207 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size": "0", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_merge_threshold": "-10", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_min_sync_interval": "0.010000", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_odsync_write": "false", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_backend": "rocksdb", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_backend_path": "", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_omap_header_cache_size": "1024", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_ondisk_finisher_threads": "1", 2024-08-06T17:50:39.208 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_thread_timeout": "60", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_op_threads": "2", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_punch_hole": "false", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_bytes": "104857600", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-08-06T17:50:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_queue_max_ops": "50", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_seek_data_hole": "false", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_sloppy_crc": "false", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_splice": "false", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_split_multiple": "2", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_split_rand_factor": "20", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_update_to": "1000", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-08-06T17:50:39.210 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_enable": "true", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_xfs_extsize": "false", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "filestore_zfs_snap": "false", 2024-08-06T17:50:39.211 INFO:teuthology.orchestra.run.smithi086.stdout: "fio_dir": "/tmp/fio", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fsid": "fd7e697e-541a-11ef-bccc-c7b262605968", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_allow_other": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_atomic_o_trunc": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_big_writes": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_debug": "false", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_default_permissions": "false", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_disable_pagecache": "false", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_max_write": "0", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_multithreaded": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_require_active_mds": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_set_user_groups": "true", 2024-08-06T17:50:39.212 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_move": "true", 2024-08-06T17:50:39.213 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_read": "true", 2024-08-06T17:50:39.213 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_splice_write": "true", 2024-08-06T17:50:39.213 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_syncfs_on_mksnap": "true", 2024-08-06T17:50:39.213 INFO:teuthology.orchestra.run.smithi086.stdout: "fuse_use_invalidate_cb": "true", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "gss_target_name": "ceph", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_file": "", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_inject_failure": "0", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "heartbeat_interval": "5", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "host": "smithi086", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-08-06T17:50:39.214 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_max_size": "1073741824", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-08-06T17:50:39.215 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "immutable_object_cache_watermark": "0.900000", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "inject_early_sigterm": "false", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "jaeger_agent_port": "6799", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "jaeger_tracing_enable": "false", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_aio": "true", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_align_min_size": "65536", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_block_align": "true", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_block_size": "4096", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_dio": "true", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_discard": "false", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_force_aio": "false", 2024-08-06T17:50:39.216 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_ignore_corruption": "false", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_max_write_bytes": "10485760", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_max_write_entries": "100", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_replay_from": "0", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_high_multiple": "0.000000", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_throttle_max_multiple": "0.000000", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_write_header_frequency": "0", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journal_zero_on_create": "false", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_prefetch_periods": "10", 2024-08-06T17:50:39.217 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_prezero_periods": "5", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "journaler_write_head_interval": "15", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "key": "", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "keyfile": "", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_backend": "rocksdb", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_default_stripe_size": "65536", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_fsck_on_mount": "false", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_fsck_on_mount_deep": "true", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_max_bytes": "67108864", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_max_ops": "512", 2024-08-06T17:50:39.218 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_nid_prealloc": "1024", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_onode_map_size": "1024", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_sync_submit_transaction": "false", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "kstore_sync_transaction": "false", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "librados_thread_count": "2", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "lockdep": "false", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "lockdep_force_backtrace": "false", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "log_coarse_timestamps": "true", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "log_flush_on_exit": "false", 2024-08-06T17:50:39.219 INFO:teuthology.orchestra.run.smithi086.stdout: "log_graylog_host": "127.0.0.1", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_graylog_port": "12201", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_max_new": "1000", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_max_recent": "10000", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_stderr_prefix": "", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_stop_at_utilization": "0.970000", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_file": "true", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_graylog": "false", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_journald": "false", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_stderr": "false", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "log_to_syslog": "false", 2024-08-06T17:50:39.220 INFO:teuthology.orchestra.run.smithi086.stdout: "max_rotating_auth_attempts": "10", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_action_on_write_error": "1", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_alternate_name_max": "8192", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_asio_thread_count": "2", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_export_pin": "true", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_dirs": "true", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_interval": "5", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_fragment_size_max": "100000", 2024-08-06T17:50:39.221 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_idle_threshold": "0.000000", 2024-08-06T17:50:39.222 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_interval": "10", 2024-08-06T17:50:39.222 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_max": "-1", 2024-08-06T17:50:39.222 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_max_until": "-1", 2024-08-06T17:50:39.222 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_merge_size": "50", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_midchunk": "0.300000", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_min_rebalance": "0.100000", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_min_start": "0.200000", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_minchunk": "0.001000", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_mode": "0", 2024-08-06T17:50:39.223 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_need_max": "1.200000", 2024-08-06T17:50:39.224 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_need_min": "0.800000", 2024-08-06T17:50:39.224 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_overload_epochs": "2", 2024-08-06T17:50:39.224 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-08-06T17:50:39.224 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_sample_interval": "3.000000", 2024-08-06T17:50:39.224 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_bits": "3", 2024-08-06T17:50:39.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_rd": "25000.000000", 2024-08-06T17:50:39.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_size": "10000", 2024-08-06T17:50:39.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_split_wr": "10000.000000", 2024-08-06T17:50:39.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_target_decay": "10.000000", 2024-08-06T17:50:39.225 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_grace": "15.000000", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_interval": "4.000000", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_beacon_mon_down_grace": "60", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_memory_limit": "4294967296", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_mid": "0.700000", 2024-08-06T17:50:39.226 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-08-06T17:50:39.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_quiesce_delay": "0", 2024-08-06T17:50:39.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_quiesce_sleep": "200", 2024-08-06T17:50:39.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_quiesce_splitauth": "true", 2024-08-06T17:50:39.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_quiesce_threshold": "524288", 2024-08-06T17:50:39.227 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_release_free_interval": "10", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_reservation": "0.050000", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_interval": "1", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cache_trim_threshold": "262144", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-08-06T17:50:39.228 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-08-06T17:50:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_delegate_inos_pct": "50", 2024-08-06T17:50:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_prealloc_inos": "1000", 2024-08-06T17:50:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-08-06T17:50:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_connect_bootstrapping": "false", 2024-08-06T17:50:39.229 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_damage_table_max_entries": "10000", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_auth_pins": "false", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_frag": "false", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_scatterstat": "false", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_debug_subtrees": "false", 2024-08-06T17:50:39.230 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_decay_halflife": "5.000000", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_default_dir_hash": "2", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_defer_session_stale": "true", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_deny_all_reconnect": "false", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_keys_per_op": "16384", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_max_commit_size": "10", 2024-08-06T17:50:39.231 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_max_entries": "0", 2024-08-06T17:50:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dir_prefetch": "true", 2024-08-06T17:50:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dirstat_min_interval": "1.000000", 2024-08-06T17:50:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_after_rejoin": "false", 2024-08-06T17:50:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_on_map": "false", 2024-08-06T17:50:39.232 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_threshold_file": "0", 2024-08-06T17:50:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-08-06T17:50:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_early_reply": "true", 2024-08-06T17:50:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_enable_op_tracker": "true", 2024-08-06T17:50:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_enforce_unique_name": "true", 2024-08-06T17:50:39.233 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_distributed": "true", 2024-08-06T17:50:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-08-06T17:50:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_random": "true", 2024-08-06T17:50:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-08-06T17:50:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-08-06T17:50:39.234 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_forward_all_requests_to_auth": "false", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_health_cache_threshold": "1.500000", 2024-08-06T17:50:39.235 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_health_summarize_threshold": "10", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_heartbeat_grace": "15.000000", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_heartbeat_reset_grace": "1000", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_health_dummy": "false", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_migrator_session_race": "false", 2024-08-06T17:50:39.236 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-08-06T17:50:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-08-06T17:50:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-08-06T17:50:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_join_fs": "", 2024-08-06T17:50:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_journal_format": "1", 2024-08-06T17:50:39.237 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-08-06T17:50:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_create_at": "0", 2024-08-06T17:50:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_dirfrag_at": "0", 2024-08-06T17:50:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_export_at": "0", 2024-08-06T17:50:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_import_at": "0", 2024-08-06T17:50:39.238 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_expire_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_journal_replay_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_link_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_mdstable_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_openc_at": "0", 2024-08-06T17:50:39.239 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_rename_at": "0", 2024-08-06T17:50:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_kill_shutdown_at": "0", 2024-08-06T17:50:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_event_large_threshold": "524288", 2024-08-06T17:50:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_events_per_segment": "1024", 2024-08-06T17:50:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_major_segment_event_ratio": "12", 2024-08-06T17:50:39.240 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_max_events": "-1", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_max_segments": "128", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_pause": "false", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_segment_size": "0", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_skip_corrupt_events": "false", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_skip_unbounded_events": "false", 2024-08-06T17:50:39.241 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-08-06T17:50:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_trim_threshold": "128", 2024-08-06T17:50:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-08-06T17:50:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_log_warn_factor": "2.000000", 2024-08-06T17:50:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_caps_per_client": "1048576", 2024-08-06T17:50:39.242 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_completed_flushes": "100000", 2024-08-06T17:50:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_completed_requests": "100000", 2024-08-06T17:50:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_export_size": "20971520", 2024-08-06T17:50:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_file_recover": "32", 2024-08-06T17:50:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_files": "64", 2024-08-06T17:50:39.243 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_ops": "8192", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_max_snaps_per_dir": "100", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_metrics_update_interval": "2", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_min_caps_per_client": "100", 2024-08-06T17:50:39.244 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_min_caps_working_set": "10000", 2024-08-06T17:50:39.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-08-06T17:50:39.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_numa_node": "-1", 2024-08-06T17:50:39.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-08-06T17:50:39.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_complaint_time": "30.000000", 2024-08-06T17:50:39.245 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_duration": "600", 2024-08-06T17:50:39.246 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_size": "20", 2024-08-06T17:50:39.246 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_slow_op_size": "20", 2024-08-06T17:50:39.246 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_history_slow_op_threshold": "10", 2024-08-06T17:50:39.246 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_op_log_threshold": "5", 2024-08-06T17:50:39.246 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_ping_grace": "15", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_ping_interval": "5", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_caps": "30000", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_max_decay_threshold": "131072", 2024-08-06T17:50:39.247 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_recall_warning_threshold": "262144", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_reconnect_timeout": "45.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_replay_interval": "1.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_root_ino_gid": "0", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_root_ino_uid": "0", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server_dispatch_client_request_delay": "0", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_server_dispatch_killpoint_random": "0.000000", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_blocklist_on_evict": "true", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_blocklist_on_timeout": "true", 2024-08-06T17:50:39.248 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_session_metadata_threshold": "16777216", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_shutdown_check": "0", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_skip_ino": "0", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_sleep_rank_change": "0.000000", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_max_uid": "4294967294", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_min_uid": "0", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_snap_rstat": "false", 2024-08-06T17:50:39.249 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_standby_replay_damaged": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_symlink_recovery": "true", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_task_status_update_interval": "2.000000", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_thrash_exports": "0", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_thrash_fragments": "0", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_tick_interval": "5.000000", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_valgrind_exit": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_verify_backtrace": "1", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_verify_scatter": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_wipe_ino_prealloc": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mds_wipe_sessions": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "mempool_debug": "false", 2024-08-06T17:50:39.250 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_debug_omit_block_device_write": "false", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_device_bytes": "1073741824", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_page_set": "false", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "memstore_page_size": "65536", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_bytes": "134217728", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_messages": "512", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_connect_retry_interval": "1.000000", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_disabled_modules": "", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_enable_op_tracker": "true", 2024-08-06T17:50:39.251 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_max_pg_creating": "1024", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_max_pg_num_change": "128", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mds_bytes": "134217728", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mds_messages": "128", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mon_bytes": "134217728", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_mon_messages": "128", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_num_op_tracker_shard": "32", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_complaint_time": "30.000000", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_history_duration": "600", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_history_size": "20", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_history_slow_op_size": "20", 2024-08-06T17:50:39.252 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_op_log_threshold": "5", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_osd_bytes": "536870912", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_osd_messages": "8192", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_pool": "true", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_service_beacon_grace": "60.000000", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_standby_modules": "true", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_stats_period": "5", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_stats_threshold": "5", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_tick_period": "2", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_accept_timeout_factor": "2.000000", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_allow_pool_delete": "true", 2024-08-06T17:50:39.253 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_allow_pool_size_one": "false", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_auth_validate_all_caps": "true", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_bytes": "104857600", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_directed_command_retry": "5", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval": "3.000000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_hunt_parallel": "3", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_log_interval": "1.000000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-08-06T17:50:39.254 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_ping_interval": "10.000000", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_ping_timeout": "30.000000", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_client_target_rank": "-1", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clock_drift_allowed": "1.000000", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_level": "debug", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_file": "true", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog": "false", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_journald": "true", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_stderr": "false", 2024-08-06T17:50:39.255 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_bootstrap": "false", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_start": "false", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_compact_on_trim": "true", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_con_tracker_persist_interval": "10", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_con_tracker_score_halflife": "43200", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_config_key_max_entry_size": "65536", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_cpu_threads": "4", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_crush_min_required_version": "hammer", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_daemon_bytes": "419430400", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-08-06T17:50:39.256 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_avail_crit": "5", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_avail_warn": "5", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_data_size_warn": "16106127360", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_block_osdmap_trim": "false", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_json": "false", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_dump_transactions": "false", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_extra_checks": "false", 2024-08-06T17:50:39.257 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_reef": "false", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_no_require_squid": "false", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_delta_reset_interval": "10.000000", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_dns_srv_name": "ceph-mon", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_down_mkfs_grace": "60", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_down_uptime_grace": "60", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_election_default_strategy": "1", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_election_timeout": "5.000000", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-08-06T17:50:39.258 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ping_divisor": "2", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_elector_ping_timeout": "2.000000", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_enable_op_tracker": "true", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_fake_pool_delete": "false", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_force_quorum_join": "false", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_fsmap_prune_threshold": "300", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_globalid_prealloc": "10000", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_detail_to_clog": "true", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_log_update_period": "5", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_max_detail": "50", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog": "true", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog_interval": "600", 2024-08-06T17:50:39.259 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_host": "[v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0]", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_host_override": "", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_initial_members": "", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_keyvaluedb": "rocksdb", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease": "5.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_full_interval": "50", 2024-08-06T17:50:39.260 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_max": "10000", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_log_max_summary": "50", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_log_entries_per_event": "4096", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_log_epochs": "500", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_mdsmap_epochs": "500", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_mgrmap_epochs": "500", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_nvmeof_epochs": "500", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_osd": "10000", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_pg_per_osd": "10000", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_pool_pg_num": "65536", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_force_trim_to": "0", 2024-08-06T17:50:39.261 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mds_skip_sanity": "false", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_memory_autotune": "true", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_memory_target": "2147483648", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_beacon_grace": "30", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_digest_period": "5", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_inactive_grace": "60", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_mkfs_grace": "240", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_min_osdmap_epochs": "50", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_nvmeofgw_beacon_grace": "10", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_nvmeofgw_set_group_id_retry": "1000", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_complaint_time": "30", 2024-08-06T17:50:39.262 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_duration": "600", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_size": "20", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_slow_op_size": "20", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_history_slow_op_threshold": "10", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_op_log_threshold": "5", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_in": "false", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_auto_mark_new_in": "true", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_cache_size": "500", 2024-08-06T17:50:39.263 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_cache_size_min": "134217728", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_crush_smoke_test": "true", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_destroyed_out_interval": "600", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_down_out_interval": "600", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_force_trim_to": "0", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_full_ratio": "0.950000", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_halflife": "3600", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_max_interval": "300", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_laggy_weight": "0.300000", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_max_initial_pgs": "1024", 2024-08-06T17:50:39.264 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_down_reporters": "2", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp": "true", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_report_timeout": "900", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_warn_num_repaired": "10", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osd_warn_op_age": "32.000000", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-08-06T17:50:39.265 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_interval": "2", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_min": "15", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_stuck_threshold": "60", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_objects": "10000", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_per_osd": "0", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pool_quota_crit_threshold": "0", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_pool_quota_warn_threshold": "0", 2024-08-06T17:50:39.266 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_probe_timeout": "2.000000", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_max_change": "0.050000", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_max_osds": "4", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_interval": "300", 2024-08-06T17:50:39.267 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_max_keys": "100", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_scrub_timeout": "300", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_session_timeout": "300", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_smart_report_timeout": "5", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stat_smooth_intervals": "6", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_pool_min_size": "2", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_pool_size": "4", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-08-06T17:50:39.268 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_subscribe_interval": "86400.000000", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_debug": "false", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_max_payload_keys": "2000", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_max_payload_size": "1048576", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_provider_kill_at": "0", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_requester_kill_at": "0", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_sync_timeout": "60.000000", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_target_pg_per_osd": "100", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_tick_interval": "5", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_timecheck_interval": "300.000000", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_use_min_delay_socket": "false", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_older_version_delay": "604800", 2024-08-06T17:50:39.269 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_filestore_osds": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_misplaced": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_older_version": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_no_app": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-08-06T17:50:39.270 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_on_too_few_osds": "false", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "monmap": "", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-08-06T17:50:39.271 INFO:teuthology.orchestra.run.smithi086.stdout: "motr_tracing_enabled": "false", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_op_threads": "3", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_buffer_size": "131072", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_cm": "false", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_device_name": "", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_dscp": "96", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_gid_idx": "0", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_local_gid": "", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_polling_us": "1000", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_port_num": "1", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-08-06T17:50:39.272 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_roce_ver": "1", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_send_buffers": "1024", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_sl": "3", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_support_srq": "true", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_rdma_type": "ib", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_async_reap_threshold": "5", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_before_connect": "false", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_ipv4": "true", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_ipv6": "false", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_msgr1": "true", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_msgr2": "true", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_port_max": "7568", 2024-08-06T17:50:39.273 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_port_min": "6800", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_prefer_ipv4": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_retry_count": "3", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_bind_retry_delay": "5", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_client": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mds": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mgr": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_mon": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_blackhole_osd": "false", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_client_mode": "crc secure", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_client_throttle_retry_time_interval": "5000", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_cluster_mode": "crc secure", 2024-08-06T17:50:39.274 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_cluster_type": "", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_compress_secure": "false", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_connection_idle_timeout": "900", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_connection_ready_timeout": "10", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_crc_data": "true", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_crc_header": "true", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_bad_msg": "false", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_bug": "true", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_old_message": "true", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_skipped_message": "false", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_die_on_unhandled_msg": "false", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_coremask": "0xF", 2024-08-06T17:50:39.275 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_devs_allowlist": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_enable_tso": "true", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hugepages": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hw_flow_control": "true", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_lro": "true", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_memory_channel": "4", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_pmd": "", 2024-08-06T17:50:39.276 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_port_id": "0", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dump_corrupt_message_level": "1", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_dump_on_send": "false", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_initial_backoff": "0.200000", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_max": "1.000000", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_probability": "0.000000", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_delay_type": "", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_internal_delays": "0.000000", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_network_congestion": "0", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_inject_socket_failures": "2500", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_learn_addr_from_peer": "true", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_max_accept_failures": "4", 2024-08-06T17:50:39.277 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_max_backoff": "15.000000", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_client_mode": "secure crc", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_cluster_mode": "secure crc", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_mon_service_mode": "secure crc", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compress_min_size": "1024", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compress_mode": "none", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_osd_compression_algorithm": "snappy", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_pq_min_cost": "65536", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_public_type": "", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_service_mode": "crc secure", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_listen_backlog": "512", 2024-08-06T17:50:39.278 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_nodelay": "true", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_tcp_rcvbuf": "0", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_time_events_min_wait_interval": "1000", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "ms_type": "async+posix", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "no_config_file": "false", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "nvmeof_mon_client_disconnect_panic": "100", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "nvmeof_mon_client_tick_period": "2", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_completion_locks_per_session": "32", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_debug_inject_relock_delay": "false", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inflight_op_bytes": "104857600", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inflight_ops": "1024", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_inject_no_watch_ping": "false", 2024-08-06T17:50:39.279 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_tick_interval": "5.000000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "objecter_timeout": "10.000000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "objectstore_blackhole": "false", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "openssl_engine_opts": "", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_delay_time": "5.000000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_hist_halflife": "1000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_max_low_ops": "2", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_max_ops": "4", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_quantize_effort": "0.100000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_agent_slop": "0.020000", 2024-08-06T17:50:39.280 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_allow_recovery_below_min_size": "true", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_asio_thread_count": "2", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_async_recovery_min_cost": "100", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_auto_mark_unfound_lost": "false", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_retry_interval": "30.000000", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_scan_max": "512", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backfill_scan_min": "64", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_degraded": "false", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_peering": "false", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_backoff_on_unfound": "true", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_beacon_report_interval": "300", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_duration": "30", 2024-08-06T17:50:39.281 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_max_block_size": "67108864", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_bench_small_size_max_iops": "100", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_blkin_trace_all": "false", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_blocked_scrub_grace_period": "120", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_check_for_log_corruption": "false", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-08-06T17:50:39.282 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_class_update_on_start": "true", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_message_cap": "256", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_message_size_cap": "524288000", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_op_priority": "63", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_client_watch_timeout": "30", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_command_max_records": "256", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_compact_on_start": "false", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_chooseleaf_type": "1", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_initial_weight": "-1.000000", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_update_on_start": "true", 2024-08-06T17:50:39.283 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_crush_update_weight_set": "true", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_drop_ping_duration": "0", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_feed_pullee": "-1", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_misdirected_ops": "false", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_no_acting_change": "false", 2024-08-06T17:50:39.284 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_no_purge_strays": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_op_order": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_pg_log_writeout": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_pretend_recovery_active": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_shutdown": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_trim_objects": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_cached_snaps": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_missing_on_start": "false", 2024-08-06T17:50:39.285 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_snaps": "false", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_keys": "1024", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_stride": "524288", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_default_data_pool_replay_window": "45", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_default_notify_timeout": "30", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep": "0.000000", 2024-08-06T17:50:39.286 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_discard_disconnected_ops": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_ec_partial_reads": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_enable_op_tracker": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_extblkdev_plugins": "vdo", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_info": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown": "true", 2024-08-06T17:50:39.287 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_fast_shutdown_timeout": "15", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_function_tracing": "false", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_grace": "20", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_interval": "6", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_peers": "10", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_min_size": "2000", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_stale": "600", 2024-08-06T17:50:39.288 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_max_size": "100000", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_min_size": "1000", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal_flush_on_shutdown": "true", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_journal_size": "5120", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_kill_backfill_at": "0", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_loop_before_reset_tphandle": "64", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_cache_size": "50", 2024-08-06T17:50:39.289 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_dedup": "true", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_message_max": "40", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_message_max_bytes": "10485760", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_map_share_max_epochs": "40", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_attr_name_len": "100", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_attr_size": "0", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_backfills": "1", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_markdown_count": "5", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_markdown_period": "600", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_name_len": "2048", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_namespace_len": "256", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_object_size": "134217728", 2024-08-06T17:50:39.290 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_omap_entries_per_request": "1024", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_blocked_by": "16", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_log_entries": "10000", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_pgls": "1024", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_push_cost": "8388608", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_push_objects": "10", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_scrubs": "3", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_trimming_pgs": "2", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_write_op_reply_len": "64", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_max_write_size": "90", 2024-08-06T17:50:39.291 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_override_recovery_settings": "false", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_profile": "balanced", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-08-06T17:50:39.292 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mclock_skip_benchmark": "false", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_base": "805306368", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_cache_min": "134217728", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target": "4294967296", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target_autotune": "false", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-08-06T17:50:39.293 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_min_pg_log_entries": "250", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_min_recovery_priority": "0", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_heartbeat_interval": "30", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_report_interval": "5", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_report_max_in_flight": "2", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_num_cache_shards": "32", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_num_op_tracker_shard": "32", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_auto_affinity": "true", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_node": "-1", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_numa_prefer_iface": "true", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-08-06T17:50:39.294 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objecter_finishers": "1", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore": "bluestore", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore_fuse": "false", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore_ideal_list_max": "64", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_objectstore_tracing": "false", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_complaint_time": "30.000000", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_duration": "600", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_size": "20", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_slow_op_size": "20", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_log_threshold": "5", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards": "0", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards_hdd": "5", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_shards_ssd": "8", 2024-08-06T17:50:39.295 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard": "0", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_pq_min_cost": "65536", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_queue": "mclock_scheduler", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_queue_cut_off": "high", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_thread_suicide_timeout": "150", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_op_thread_timeout": "15", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_open_classes_on_start": "true", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_os_flags": "0", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_peering_op_priority": "255", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_delete_cost": "1048576", 2024-08-06T17:50:39.296 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_delete_priority": "5", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_dups_tracked": "3000", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_trim_max": "10000", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_log_trim_min": "100", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_object_context_cache_count": "64", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-08-06T17:50:39.297 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_crimson": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_crush_rule": "-1", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_ec_fast_read": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_bulk": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nodelete": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_flags": "0", 2024-08-06T17:50:39.298 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_min_size": "2", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pg_num": "32", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_pgp_num": "0", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_read_ratio": "70", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_size": "2", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_default_type": "replicated", 2024-08-06T17:50:39.299 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_pool_use_gmt_hitset": "true", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_push_per_object_cost": "1000", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_read_ec_check_for_errors": "false", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recover_clone_overlap": "true", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recover_clone_overlap_limit": "10", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_cost": "20971520", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_delay_start": "0.000000", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active": "0", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active_hdd": "3", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_active_ssd": "10", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_chunk": "8388608", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-08-06T17:50:39.300 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_max_single_start": "1", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_op_priority": "3", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_op_warn_multiple": "16", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_priority": "5", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_retry_interval": "30.000000", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep": "0.000000", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_repair_during_recovery": "false", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_requested_scrub_priority": "120", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-08-06T17:50:39.301 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_rollback_to_cluster_snap": "", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_auto_repair": "false", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_begin_hour": "0", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_begin_week_day": "0", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_chunk_max": "25", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_chunk_min": "5", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_cost": "52428800", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_during_recovery": "false", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_end_hour": "0", 2024-08-06T17:50:39.302 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_end_week_day": "0", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_event_cost": "4096", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_invalid_stats": "true", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_load_threshold": "0.500000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_max_interval": "604800.000000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_max_preemptions": "5", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_min_interval": "86400.000000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_priority": "5", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_scrub_sleep": "0.000000", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-08-06T17:50:39.303 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_shutdown_pgref_assert": "false", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_skip_check_past_interval_bounds": "false", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_skip_data_digest": "false", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_smart_report_timeout": "5", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_cost": "1048576", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_priority": "5", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep": "0.000000", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_stats_update_period_scrubbing": "15", 2024-08-06T17:50:39.304 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_target_transaction_size": "30", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_default_cache_mode": "writeback", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_tracing": "false", 2024-08-06T17:50:39.305 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_use_stale_snap": "false", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "osdc_blkin_trace_all": "false", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_kill_at": "0", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_max_join_drift": "10", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_min": "500", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_min_wait": "0.050000", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_propose_interval": "1.000000", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_max": "500", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_max_multiplier": "20", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_service_trim_min": "10", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_stash_full_interval": "25", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_trim_max": "500", 2024-08-06T17:50:39.306 INFO:teuthology.orchestra.run.smithi086.stdout: "paxos_trim_min": "250", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "perf": "true", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "pid_file": "", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addr": "-", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_addrv": "", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_bind_addr": "-", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_network": "", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "public_network_interface": "", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "qat_compressor_busy_polling": "false", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "qat_compressor_enabled": "false", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "qat_compressor_session_max_number": "256", 2024-08-06T17:50:39.307 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_mon_op_timeout": "0", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_osd_op_timeout": "0", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rados_tracing": "false", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_atime_update_interval": "60", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_balance_parent_reads": "false", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_balance_snap_reads": "false", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blkin_trace_all": "false", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blocklist_expire_seconds": "0", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_blocklist_on_break_lock": "true", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache": "true", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_block_writes_upfront": "false", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty": "25165824", 2024-08-06T17:50:39.308 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_max_dirty_object": "0", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_policy": "writearound", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_size": "33554432", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_target_dirty": "16777216", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_clone_copy_on_read": "false", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_compression_hint": "none", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_concurrent_management_ops": "10", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_clone_format": "auto", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_data_pool": "", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_features": "61", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_format": "2", 2024-08-06T17:50:39.309 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_map_options": "", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_order": "22", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_pool": "rbd", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_stripe_count": "0", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_default_stripe_unit": "0", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_disable_zero_copy_writes": "true", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_discard_granularity_bytes": "65536", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_enable_alloc_hint": "true", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_io_scheduler": "simple", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-08-06T17:50:39.310 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_commit_age": "5.000000", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_flush_interval": "0", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_order": "24", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_pool": "", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_journal_splay_width": "4", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_localize_parent_reads": "false", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_localize_snap_reads": "false", 2024-08-06T17:50:39.311 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_type": "simple", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-08-06T17:50:39.312 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_autotune": "true", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_base": "805306368", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_memory_target": "4294967296", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_delete_delay": "0", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-08-06T17:50:39.313 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_replay_delay": "0", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_to_trash_on_remove": "false", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_mtime_update_interval": "60", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_non_blocking_aio": "true", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_op_thread_timeout": "60", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_op_threads": "1", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_parent_cache_enabled": "false", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_mode": "disabled", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_path": "/tmp", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_persistent_cache_size": "1073741824", 2024-08-06T17:50:39.314 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_plugins": "", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_burst": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_bps_limit": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_exclude_ops": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_burst": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_iops_limit": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_burst": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_bps_limit": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_burst": "0", 2024-08-06T17:50:39.315 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_read_iops_limit": "0", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_schedule_tick_min": "50", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_burst": "0", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_bps_limit": "0", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_burst": "0", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_qos_write_iops_limit": "0", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_quiesce_notification_attempts": "10", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_read_from_replica_policy": "default", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-08-06T17:50:39.316 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_max_bytes": "524288", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_readahead_trigger_requests": "10", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_request_timed_out_seconds": "30", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_skip_partial_discard": "true", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_tracing": "false", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_validate_names": "true", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rbd_validate_pool": "true", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "restapi_base_url": "", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "restapi_log_level": "", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_account_default_quota_max_size": "-1", 2024-08-06T17:50:39.317 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_acl_grants_max_num": "100", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_admin_entry": "admin", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_asio_assert_yielding": "false", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_backend_store": "rados", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_barbican_url": "", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_beast_enable_async": "true", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_counters_cache": "false", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_index_max_aio": "128", 2024-08-06T17:50:39.318 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_quota_ttl": "600", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_enabled": "true", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_expiry_interval": "900", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cache_lru_size": "25000", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_config_store": "rados", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_content_length_compat": "false", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_copy_obj_progress": "true", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cors_rules_max_num": "100", 2024-08-06T17:50:39.319 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_cross_domain_policy": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_default_encryption_key": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_addr": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_ca_path": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_client_cert": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_client_key": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_password": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_kmip_username": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_require_ssl": "true", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-08-06T17:50:39.320 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-08-06T17:50:39.321 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_suppress_logs": "true", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_addr": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_auth": "token", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_namespace": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_prefix": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_token_file": "", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_buffersize": "524288", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_low_speed_limit": "1024", 2024-08-06T17:50:39.322 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_low_speed_time": "300", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_tcp_keepalive": "0", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_fadvise": "4", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_address": "127.0.0.1:6379", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_l1_datacache_address": "127.0.0.1:6379", 2024-08-06T17:50:39.323 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_l1_datacache_persistent_path": "/tmp/rgw_d4n_datacache/", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_l1_datacache_size": "1073741824", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_l1_evict_cache_on_start": "true", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_l1_fadvise": "4", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_libaio_aio_num": "64", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_d4n_libaio_aio_threads": "20", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_changes_size": "1000", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_num_shards": "128", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_log_window": "30", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_notify_interval_msec": "0", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_sync_poll_interval": "20", 2024-08-06T17:50:39.324 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_data_sync_spawn_window": "20", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_data_log_backing": "fifo", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_region_info_oid": "default.region", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_defer_to_bucket_acls": "", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_disable_s3select": "false", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-08-06T17:50:39.325 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_res": "500.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dns_name": "", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dns_s3website_name": "", 2024-08-06T17:50:39.326 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dynamic_resharding": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dynamic_resharding_may_reduce": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_dynamic_resharding_reduction_wait": "120", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_gc_threads": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_lc_threads": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_mdsearch": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_ops_log": "false", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_quota_threads": "true", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_static_website": "false", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enable_usage_log": "false", 2024-08-06T17:50:39.327 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_enforce_swift_acls": "true", 2024-08-06T17:50:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_exit_timeout_secs": "120", 2024-08-06T17:50:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_expose_bucket": "false", 2024-08-06T17:50:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_extended_http_attrs": "", 2024-08-06T17:50:39.328 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_filter": "none", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_frontends": "beast port=7480", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_concurrent_io": "10", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_deferred": "50", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_objs": "32", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_queue_size": "134213632", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_max_trim_chunk": "16", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_obj_min_wait": "7200", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_processor_max_time": "3600", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_gc_processor_period": "3600", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-08-06T17:50:39.329 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_get_obj_window_size": "16777216", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_healthcheck_disabling_path": "", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ignore_get_invalid_range": "false", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_init_timeout": "300", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_kafka_connection_idle": "300", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_kafka_message_timeout": "5000", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_kafka_sleep_timeout": "10", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-08-06T17:50:39.330 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_domain": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_password": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_password_path": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_project": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_tenant": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_token": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_token_path": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_admin_user": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_api_version": "2", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_domain": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_password": "", 2024-08-06T17:50:39.331 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_project": "", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_tenant": "", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_barbican_user": "", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_implicit_tenants": "false", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_service_token_enabled": "false", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_token_cache_size": "10000", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_url": "", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_keystone_verify_ssl": "true", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_debug_interval": "-1", 2024-08-06T17:50:39.332 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_lock_max_time": "90", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_objs": "32", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_rules": "1000", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_worker": "3", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_max_wp_worker": "3", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lc_thread_delay": "0", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_dnattr": "uid", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_searchfilter": "", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ldap_uri": "", 2024-08-06T17:50:39.333 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lfuda_sync_frequency": "60", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_http_headers": "", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_nonexistent_bucket": "false", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_log_object_name_utc": "false", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attr_name_len": "0", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attr_size": "0", 2024-08-06T17:50:39.334 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_attrs_num_in_req": "0", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_chunk_size": "4194304", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_concurrent_requests": "1024", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_dynamic_shards": "1999", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_listing_results": "1000", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_notify_retries": "10", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_objs_per_shard": "100000", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_put_param_size": "1048576", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_put_size": "5368709120", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_max_slo_entries": "1000", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_md_log_max_shards": "64", 2024-08-06T17:50:39.335 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_md_notify_interval_msec": "200", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_meta_sync_poll_interval": "20", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_meta_sync_spawn_window": "20", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_mp_lock_max_time": "600", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multipart_min_part_size": "5242880", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_fhcache_size": "2017", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_lru_lanes": "5", 2024-08-06T17:50:39.336 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_max_gc": "300", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_gc_threads": "false", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_lc_threads": "false", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_quota_threads": "false", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_run_sync_thread": "false", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_num_async_rados_threads": "32", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_num_control_oids": "8", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_numa_node": "-1", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_obj_stripe_size": "4194304", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-08-06T17:50:39.337 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_chunk_size": "100", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_gc_interval": "600", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_objexp_hints_num_shards": "127", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_thread_timeout": "600", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_op_tracing": "false", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_token": "", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_url": "", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_opa_verify_ssl": "true", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-08-06T17:50:39.338 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_rados": "false", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_ops_log_socket_path": "", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_push_interval": "2.000000", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_push_interval_max": "30.000000", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_period_root_pool": ".rgw.root", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_cache_lanes": "3", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-08-06T17:50:39.339 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_cache_max_buckets": "100", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_cache_partitions": "3", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_print_continue": "true", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_print_prohibited_content_length": "false", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rados_tracing": "false", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm": "", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm_id": "", 2024-08-06T17:50:39.340 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_region": "", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_region_root_pool": ".rgw.root", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_relaxed_region_enforcement": "false", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_request_uri": "", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_batch_size": "64", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_debug_interval": "-1", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_max_aio": "128", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_num_logs": "16", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_reshard_thread_interval": "600", 2024-08-06T17:50:39.341 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_resolve_cname": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_rest_getusage_op_compat": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_run_sync_thread": "true", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_keystone": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_ldap": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_rados": "true", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_auth_use_sts": "false", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_s3_success_create_obj_status": "0", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_scheduler_type": "throttler", 2024-08-06T17:50:39.342 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_script_uri": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_service_provider_name": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_client_id": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_client_secret": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_entry": "sts", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_key": "sts", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_max_session_duration": "43200", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_min_session_duration": "900", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sts_token_introspection_url": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_account_in_url": "false", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_auth_entry": "auth", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_auth_url": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_custom_header": "", 2024-08-06T17:50:39.343 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_enforce_content_length": "false", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_need_stats": "true", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_tenant_name": "", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_token_expiration": "86400", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_url": "", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_url_prefix": "swift", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_swift_versioning_enabled": "false", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_lease_period": "120", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_interval": "1200", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-08-06T17:50:39.344 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_obj_etag_verify": "false", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_history_size": "4096", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_thread_pool_size": "512", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_topic_persistency_max_retries": "0", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-08-06T17:50:39.345 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_topic_require_publish_policy": "false", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_comment": "", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_createby": "", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_encoding": "", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_flag": "false", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_max_size": "5368709120", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_origin": "", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_sha_unit": "524288", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_torrent_tracker": "", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_trust_forwarded_https": "false", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_log_tick_interval": "30", 2024-08-06T17:50:39.346 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_max_shards": "32", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_usage_max_user_shards": "1", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_use_opa_authz": "false", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_counters_cache": "false", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_counters_cache_size": "10000", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_default_quota_max_size": "-1", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_max_buckets": "1000", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_policies_max_num": "100", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_interval": "86400", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-08-06T17:50:39.347 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_user_unique_email": "true", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_verify_ssl": "true", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_website_routing_rules_max_num": "50", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone": "", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone_id": "", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup": "", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup_id": "", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgwlc_auto_session_clear": "true", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rgwlc_skip_bucket_step": "false", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_block_size": "4096", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-08-06T17:50:39.348 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_shard_bits": "4", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_size": "536870912", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cache_type": "binned_lru", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_compaction_stats": "false", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_extended_stats": "false", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_collect_memory_stats": "false", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_index_type": "binary_search", 2024-08-06T17:50:39.349 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_log_to_ceph_log": "true", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_metadata_block_size": "4096", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_partition_filters": "false", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_perf": "false", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "rotating_keys_renewal_timeout": "10", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "run_dir": "/var/run/ceph", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_block_create": "true", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_cache_lru_size": "67108864", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_cbjournal_size": "5368709120", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_data_delta_based_overwrite": "0", 2024-08-06T17:50:39.350 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_default_max_object_size": "16777216", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_device_size": "53687091200", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_full_integrity_check": "false", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_capacity": "16", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_journal_iodepth_limit": "5", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_main_device_type": "SSD", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_max_concurrent_transactions": "8", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_max_data_allocation_size": "32768", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-08-06T17:50:39.351 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "seastore_segment_size": "67108864", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "set_keepcaps": "false", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "setgroup": "ceph", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "setuser": "ceph", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "setuser_match_path": "", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "target_max_misplaced_ratio": "0.050000", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "thp": "false", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "threadpool_default_timeout": "60", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "threadpool_empty_queue_max_wait": "2", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "throttler_perf_counter": "true", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "tmp_dir": "/tmp", 2024-08-06T17:50:39.352 INFO:teuthology.orchestra.run.smithi086.stdout: "tmp_file_template": "/tmp/ceph-mon.a.XXXXXX" 2024-08-06T17:50:39.353 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:50:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:39 smithi086 ceph-mon[29229]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:39.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-08-06T17:50:39.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:39 smithi107 ceph-mon[32381]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:39 smithi117 ceph-mon[32835]: pgmap v314: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:40.204 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:41.935 INFO:teuthology.orchestra.run.smithi086.stdout:mon.a: {} 2024-08-06T17:50:41.936 INFO:teuthology.orchestra.run.smithi086.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-06T17:50:41.944 INFO:teuthology.orchestra.run.smithi086.stdout:mon.c: {} 2024-08-06T17:50:41.944 INFO:teuthology.orchestra.run.smithi086.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-06T17:50:41.954 INFO:teuthology.orchestra.run.smithi086.stdout:mon.b: {} 2024-08-06T17:50:41.954 INFO:teuthology.orchestra.run.smithi086.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-08-06T17:50:42.652 INFO:teuthology.run_tasks:Running task exec... 2024-08-06T17:50:42.665 INFO:teuthology.task.exec:Executing custom commands... 2024-08-06T17:50:42.665 INFO:teuthology.task.exec:Running commands on role client.2 host ubuntu@smithi117.front.sepia.ceph.com 2024-08-06T17:50:42.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror' 2024-08-06T17:50:42.666 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-08-06T17:50:42.666 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-06T17:50:42.666 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-06T17:50:42.666 INFO:tasks.thrashosds.thrasher:Killing osd 3, live_osds are [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-06T17:50:42.666 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-08-06T17:50:42.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:50:42.667 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-08-06T17:50:42.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:42.668 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-08-06T17:50:42.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:50:42.668 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-08-06T17:50:42.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T17:50:42.669 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-08-06T17:50:42.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:50:42.708 INFO:teuthology.orchestra.run.smithi117.stdout:Running main() from gmock_main.cc 2024-08-06T17:50:42.708 INFO:teuthology.orchestra.run.smithi117.stdout:Note: Google Test filter = -TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror 2024-08-06T17:50:42.708 INFO:teuthology.orchestra.run.smithi117.stdout:[==========] Running 68 tests from 1 test suite. 2024-08-06T17:50:42.708 INFO:teuthology.orchestra.run.smithi117.stdout:[----------] Global test environment set-up. 2024-08-06T17:50:42.708 INFO:teuthology.orchestra.run.smithi117.stdout:[----------] 68 tests from TestClsRbd 2024-08-06T17:50:42.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:42.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:42.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:42 smithi107 ceph-mon[32381]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1442883866' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:50:42.970 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:42 smithi117 ceph-mon[32835]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:42.970 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1442883866' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:50:42.970 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sent signal SIGHUP to main process 54716 (conmon) on client request. 2024-08-06T17:50:42.970 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sending signal SIGHUP to process 54718 (podman-init) on client request. 2024-08-06T17:50:42.970 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sending signal SIGHUP to process 54720 (ceph-osd) on client request. 2024-08-06T17:50:43.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:43.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:42 smithi086 ceph-mon[29229]: pgmap v315: 1 pgs: 1 active+clean; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:43.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1442883866' entity='client.admin' cmd={"prefix": "mon dump", "format": "json"} : dispatch 2024-08-06T17:50:43.116 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:42 smithi086 systemd[1]: Stopping Ceph osd.3 for fd7e697e-541a-11ef-bccc-c7b262605968... 2024-08-06T17:50:43.117 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sent signal SIGHUP to main process 49377 (conmon) on client request. 2024-08-06T17:50:43.117 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sending signal SIGHUP to process 49379 (podman-init) on client request. 2024-08-06T17:50:43.117 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sending signal SIGHUP to process 49381 (ceph-osd) on client request. 2024-08-06T17:50:43.180 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:43.183 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:43.206 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:43.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:43.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:43.373 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sent signal SIGHUP to main process 37670 (conmon) on client request. 2024-08-06T17:50:43.373 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 37672 (podman-init) on client request. 2024-08-06T17:50:43.373 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:42 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 37674 (ceph-osd) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sent signal SIGHUP to main process 36932 (conmon) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sending signal SIGHUP to process 36934 (podman-init) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sending signal SIGHUP to process 36936 (ceph-osd) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sent signal SIGHUP to main process 42803 (conmon) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 42805 (podman-init) on client request. 2024-08-06T17:50:43.376 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 42807 (ceph-osd) on client request. 2024-08-06T17:50:43.480 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[60694]: 2024-08-06T17:50:43.163+0000 7f2b4cadc640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-08-06T17:50:43.480 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[60694]: 2024-08-06T17:50:43.163+0000 7f2b4cadc640 -1 osd.3 98 *** Got signal Terminated *** 2024-08-06T17:50:43.480 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[60694]: 2024-08-06T17:50:43.163+0000 7f2b4cadc640 -1 osd.3 98 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-08-06T17:50:43.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:43.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:43.670 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sent signal SIGHUP to control process 59046 (bash) on client request. 2024-08-06T17:50:43.670 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 59052 (podman) on client request. 2024-08-06T17:50:43.670 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 42805 (podman-init) on client request. 2024-08-06T17:50:43.670 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 42807 (ceph-osd) on client request. 2024-08-06T17:50:43.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:43.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.1 marked itself down and dead 2024-08-06T17:50:43.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.11 marked itself down and dead 2024-08-06T17:50:43.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.3 marked itself down and dead 2024-08-06T17:50:43.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.8 marked itself down and dead 2024-08-06T17:50:43.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.4 marked itself down and dead 2024-08-06T17:50:43.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:43 smithi117 ceph-mon[32835]: osd.5 marked itself down and dead 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sent signal SIGHUP to main process 43346 (conmon) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 43348 (podman-init) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 43350 (ceph-osd) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sent signal SIGHUP to control process 58835 (bash) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 37672 (podman-init) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 37674 (ceph-osd) on client request. 2024-08-06T17:50:43.776 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 58881 (podman) on client request. 2024-08-06T17:50:43.777 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 podman[58956]: 2024-08-06 17:50:43.726245485 +0000 UTC m=+0.057795661 container stop 08d4169833c6ce750d11efc036448f00c6c5c79f8f33e5e39327a5540199168d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-8, RELEASE=main-93ef537, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0) 2024-08-06T17:50:43.777 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 podman[58956]: 2024-08-06 17:50:43.726740391 +0000 UTC m=+0.058290565 container died 08d4169833c6ce750d11efc036448f00c6c5c79f8f33e5e39327a5540199168d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-8, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-06T17:50:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.1 marked itself down and dead 2024-08-06T17:50:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.11 marked itself down and dead 2024-08-06T17:50:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.3 marked itself down and dead 2024-08-06T17:50:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.8 marked itself down and dead 2024-08-06T17:50:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.4 marked itself down and dead 2024-08-06T17:50:43.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:43 smithi086 ceph-mon[29229]: osd.5 marked itself down and dead 2024-08-06T17:50:43.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:43.926 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sent signal SIGHUP to control process 59008 (bash) on client request. 2024-08-06T17:50:43.927 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sending signal SIGHUP to process 59060 (podman) on client request. 2024-08-06T17:50:43.927 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sending signal SIGHUP to process 36934 (podman-init) on client request. 2024-08-06T17:50:43.927 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Sending signal SIGHUP to process 36936 (ceph-osd) on client request. 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.1 marked itself down and dead 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.11 marked itself down and dead 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.3 marked itself down and dead 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.8 marked itself down and dead 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.4 marked itself down and dead 2024-08-06T17:50:43.927 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:43 smithi107 ceph-mon[32381]: osd.5 marked itself down and dead 2024-08-06T17:50:44.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:44.080 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sent signal SIGHUP to control process 58910 (bash) on client request. 2024-08-06T17:50:44.080 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Sending signal SIGHUP to process 58956 (podman) on client request. 2024-08-06T17:50:44.080 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Deactivated successfully. 2024-08-06T17:50:44.080 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Unit process 58956 (podman) remains running after unit stopped. 2024-08-06T17:50:44.080 INFO:journalctl@ceph.osd.8.smithi117.stdout:Aug 06 17:50:43 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8.service: Consumed 1.798s CPU time. 2024-08-06T17:50:44.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:44.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:44.299 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sent signal SIGHUP to main process 55035 (conmon) on client request. 2024-08-06T17:50:44.299 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:44.300 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:44.373 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:44 smithi117 podman[58853]: 2024-08-06 17:50:44.081882866 +0000 UTC m=+1.068026975 container stop d06c1494c2ec32d72fd443b2b763ea3e6b5bd81315c463eadd9c77604209df7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux ) 2024-08-06T17:50:44.373 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:44 smithi117 podman[58853]: 2024-08-06 17:50:44.081998904 +0000 UTC m=+1.068143003 container died d06c1494c2ec32d72fd443b2b763ea3e6b5bd81315c463eadd9c77604209df7f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7) 2024-08-06T17:50:44.389 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sent signal SIGHUP to main process 48448 (conmon) on client request. 2024-08-06T17:50:44.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:44.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:44.390 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 podman[59125]: 2024-08-06 17:50:44.036132335 +0000 UTC m=+0.494967928 container stop b43d612986a5704f3b9d859d01d9fa8c668445c3796120f2e9cebc0ee5c55f0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-93ef537) 2024-08-06T17:50:44.390 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 podman[59125]: 2024-08-06 17:50:44.03798164 +0000 UTC m=+0.496817217 container died b43d612986a5704f3b9d859d01d9fa8c668445c3796120f2e9cebc0ee5c55f0e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-08-06T17:50:44.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:44.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:44.616 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:44 smithi086 podman[71886]: 2024-08-06 17:50:44.301231786 +0000 UTC m=+1.268637428 container died 56ed2a88f7c4ae34c1fd66cae85f872e443bcb80933138e1195a4e6651095061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, RELEASE=main-93ef537, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True) 2024-08-06T17:50:44.616 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sent signal SIGHUP to control process 71755 (bash) on client request. 2024-08-06T17:50:44.616 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sending signal SIGHUP to process 71807 (podman) on client request. 2024-08-06T17:50:44.634 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:44 smithi117 bash[58853]: Error: container d06c1494c2ec32d72fd443b2b763ea3e6b5bd81315c463eadd9c77604209df7f conmon exited prematurely, exit code could not be retrieved: internal libpod error 2024-08-06T17:50:44.634 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sent signal SIGHUP to control process 58953 (bash) on client request. 2024-08-06T17:50:44.634 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 43348 (podman-init) on client request. 2024-08-06T17:50:44.634 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 43350 (ceph-osd) on client request. 2024-08-06T17:50:44.634 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 59016 (podman) on client request. 2024-08-06T17:50:44.647 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sent signal SIGHUP to main process 54364 (conmon) on client request. 2024-08-06T17:50:44.647 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 54366 (podman-init) on client request. 2024-08-06T17:50:44.647 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 54369 (ceph-osd) on client request. 2024-08-06T17:50:44.681 INFO:teuthology.orchestra.run.smithi117.stdout:[ RUN ] TestClsRbd.get_all_features 2024-08-06T17:50:44.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:44.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:44.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:44.963 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 bash[59125]: Error: container b43d612986a5704f3b9d859d01d9fa8c668445c3796120f2e9cebc0ee5c55f0e conmon exited prematurely, exit code could not be retrieved: internal libpod error 2024-08-06T17:50:44.963 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sent signal SIGHUP to control process 59097 (bash) on client request. 2024-08-06T17:50:44.963 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Sending signal SIGHUP to process 59300 (podman) on client request. 2024-08-06T17:50:44.964 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Deactivated successfully. 2024-08-06T17:50:44.964 INFO:journalctl@ceph.osd.5.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5.service: Consumed 2.070s CPU time. 2024-08-06T17:50:44.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sent signal SIGHUP to control process 59211 (bash) on client request. 2024-08-06T17:50:44.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:44.965 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:44.965 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 59212 (podman) on client request. 2024-08-06T17:50:44.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:44.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: Health check failed: 6 osds down (OSD_DOWN) 2024-08-06T17:50:44.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: osdmap e99: 12 total, 6 up, 12 in 2024-08-06T17:50:44.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.117:0/2463228055' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:44.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:44.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:44 smithi107 ceph-mon[32381]: osd.9 marked itself down and dead 2024-08-06T17:50:44.966 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:44 smithi107 podman[59315]: 2024-08-06 17:50:44.753950765 +0000 UTC m=+0.025694534 container stop 2b576a9c6e64314678335089eed60d8075e8b2df27e63e8a70ac9db56aaf1e14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, ceph=True, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-08-06T17:50:44.966 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:44 smithi107 podman[59315]: 2024-08-06 17:50:44.754352348 +0000 UTC m=+0.026096107 container died 2b576a9c6e64314678335089eed60d8075e8b2df27e63e8a70ac9db56aaf1e14 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, RELEASE=main-93ef537, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, ceph=True) 2024-08-06T17:50:45.013 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:44 smithi086 podman[71886]: 2024-08-06 17:50:44.822208303 +0000 UTC m=+1.789613941 container cleanup 56ed2a88f7c4ae34c1fd66cae85f872e443bcb80933138e1195a4e6651095061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=main-93ef537, GIT_BRANCH=HEAD) 2024-08-06T17:50:45.013 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:44 smithi086 bash[71886]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: Health check failed: 6 osds down (OSD_DOWN) 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: osdmap e99: 12 total, 6 up, 12 in 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.117:0/2463228055' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:45.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:44 smithi086 ceph-mon[29229]: osd.9 marked itself down and dead 2024-08-06T17:50:45.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:45.123 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:44 smithi117 bash[59120]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11 2024-08-06T17:50:45.123 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 podman[59096]: 2024-08-06 17:50:44.900148194 +0000 UTC m=+0.140632598 container stop ed91351e8ad9054c90571a0846a69f95be82ec57f42d9734f7cc555c06b28a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-9, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0) 2024-08-06T17:50:45.123 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:44 smithi117 podman[59096]: 2024-08-06 17:50:44.900720344 +0000 UTC m=+0.141204754 container died ed91351e8ad9054c90571a0846a69f95be82ec57f42d9734f7cc555c06b28a37 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-9, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731) 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: pgmap v316: 1 pgs: 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: Health check failed: 6 osds down (OSD_DOWN) 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: osdmap e99: 12 total, 6 up, 12 in 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.117:0/2463228055' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true} : dispatch 2024-08-06T17:50:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:44 smithi117 ceph-mon[32835]: osd.9 marked itself down and dead 2024-08-06T17:50:45.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:45.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:45.366 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:45 smithi086 podman[72024]: 2024-08-06 17:50:45.015191874 +0000 UTC m=+0.843491304 container remove 56ed2a88f7c4ae34c1fd66cae85f872e443bcb80933138e1195a4e6651095061 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, RELEASE=main-93ef537, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731) 2024-08-06T17:50:45.431 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 bash[59315]: Error: container 2b576a9c6e64314678335089eed60d8075e8b2df27e63e8a70ac9db56aaf1e14 conmon exited prematurely, exit code could not be retrieved: internal libpod error 2024-08-06T17:50:45.431 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sent signal SIGHUP to control process 59350 (bash) on client request. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 59351 (podman) on client request. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Deactivated successfully. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Unit process 48450 (podman-init) remains running after unit stopped. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Unit process 48452 (ceph-osd) remains running after unit stopped. 2024-08-06T17:50:45.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Consumed 1.743s CPU time. 2024-08-06T17:50:45.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:45.512 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:45 smithi117 bash[59096]: Error: container ed91351e8ad9054c90571a0846a69f95be82ec57f42d9734f7cc555c06b28a37 conmon exited prematurely, exit code could not be retrieved: internal libpod error 2024-08-06T17:50:45.512 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:45 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sent signal SIGHUP to control process 59055 (bash) on client request. 2024-08-06T17:50:45.512 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:45 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Sending signal SIGHUP to process 59165 (podman) on client request. 2024-08-06T17:50:45.512 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:45 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Deactivated successfully. 2024-08-06T17:50:45.512 INFO:journalctl@ceph.osd.9.smithi117.stdout:Aug 06 17:50:45 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9.service: Consumed 1.771s CPU time. 2024-08-06T17:50:45.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:45.683 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:45.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:45.684 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.33667611 +0000 UTC m=+0.019190655 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:50:45.684 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.43641112 +0000 UTC m=+0.118925650 container create 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0) 2024-08-06T17:50:45.747 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:45.769 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:45 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-06T17:50:45.769 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:45 smithi117 ceph-mon[32835]: osdmap e100: 12 total, 5 up, 12 in 2024-08-06T17:50:45.769 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:45 smithi117 ceph-mon[32835]: osd.9 marked itself dead as of e100 2024-08-06T17:50:45.769 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:45 smithi117 ceph-mon[32835]: pgmap v319: 33 pgs: 3 undersized+peered, 29 unknown, 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:45.769 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:45 smithi117 ceph-mon[32835]: osd.7 marked itself down and dead 2024-08-06T17:50:45.769 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.513842689 +0000 UTC m=+0.102806671 container create 58a1213b7dee7493f5409476f69dce86caef45522e7386b66cf154a4f5fd061c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=main-93ef537, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-08-06T17:50:45.770 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.427935754 +0000 UTC m=+0.016899753 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:50:45.770 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.725472936 +0000 UTC m=+0.314436924 container init 58a1213b7dee7493f5409476f69dce86caef45522e7386b66cf154a4f5fd061c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-93ef537, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=) 2024-08-06T17:50:45.770 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.729188426 +0000 UTC m=+0.318152417 container start 58a1213b7dee7493f5409476f69dce86caef45522e7386b66cf154a4f5fd061c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11-deactivate, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, RELEASE=main-93ef537, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-08-06T17:50:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:45 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-06T17:50:45.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:45 smithi086 ceph-mon[29229]: osdmap e100: 12 total, 5 up, 12 in 2024-08-06T17:50:45.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:45 smithi086 ceph-mon[29229]: osd.9 marked itself dead as of e100 2024-08-06T17:50:45.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:45 smithi086 ceph-mon[29229]: pgmap v319: 33 pgs: 3 undersized+peered, 29 unknown, 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:45.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:45 smithi086 ceph-mon[29229]: osd.7 marked itself down and dead 2024-08-06T17:50:45.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sent signal SIGHUP to main process 43625 (conmon) on client request. 2024-08-06T17:50:45.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:45.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:45.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:45.963 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sent signal SIGHUP to control process 59243 (bash) on client request. 2024-08-06T17:50:45.963 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 59470 (podman) on client request. 2024-08-06T17:50:45.963 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 54366 (podman-init) on client request. 2024-08-06T17:50:45.963 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 54369 (ceph-osd) on client request. 2024-08-06T17:50:45.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:45 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "test-rados-api-smithi117-58763-1","app": "rados","yes_i_really_mean_it": true}]': finished 2024-08-06T17:50:45.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:45 smithi107 ceph-mon[32381]: osdmap e100: 12 total, 5 up, 12 in 2024-08-06T17:50:45.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:45 smithi107 ceph-mon[32381]: osd.9 marked itself dead as of e100 2024-08-06T17:50:45.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:45 smithi107 ceph-mon[32381]: pgmap v319: 33 pgs: 3 undersized+peered, 29 unknown, 1 active+clean; 577 KiB data, 325 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:45.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:45 smithi107 ceph-mon[32381]: osd.7 marked itself down and dead 2024-08-06T17:50:45.964 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.799213756 +0000 UTC m=+0.481728290 container init 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0) 2024-08-06T17:50:45.964 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.802754747 +0000 UTC m=+0.485269283 container start 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, RELEASE=main-93ef537, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-08-06T17:50:45.965 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.890079137 +0000 UTC m=+0.572593677 container attach 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=main-93ef537, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, io.buildah.version=1.36.0) 2024-08-06T17:50:46.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:46.034 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.809063761 +0000 UTC m=+0.398027756 container attach 58a1213b7dee7493f5409476f69dce86caef45522e7386b66cf154a4f5fd061c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-93ef537, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-08-06T17:50:46.034 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:45 smithi117 podman[59220]: 2024-08-06 17:50:45.927058169 +0000 UTC m=+0.516022163 container died 58a1213b7dee7493f5409476f69dce86caef45522e7386b66cf154a4f5fd061c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-11-deactivate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=main-93ef537, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-08-06T17:50:46.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:46.271 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:45 smithi107 podman[59452]: 2024-08-06 17:50:45.997706075 +0000 UTC m=+0.680220603 container died 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0) 2024-08-06T17:50:46.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:46.373 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sent signal SIGHUP to control process 59066 (bash) on client request. 2024-08-06T17:50:46.373 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sending signal SIGHUP to process 59220 (podman) on client request. 2024-08-06T17:50:46.374 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sending signal SIGHUP to process 59235 (conmon) on client request. 2024-08-06T17:50:46.374 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Sending signal SIGHUP to process 59261 (podman) on client request. 2024-08-06T17:50:46.374 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Deactivated successfully. 2024-08-06T17:50:46.374 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Unit process 59220 (podman) remains running after unit stopped. 2024-08-06T17:50:46.374 INFO:journalctl@ceph.osd.11.smithi117.stdout:Aug 06 17:50:46 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11.service: Consumed 1.587s CPU time. 2024-08-06T17:50:46.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:46.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:46.573 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sent signal SIGHUP to control process 72078 (bash) on client request. 2024-08-06T17:50:46.573 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Sending signal SIGHUP to process 72079 (podman) on client request. 2024-08-06T17:50:46.574 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Deactivated successfully. 2024-08-06T17:50:46.574 INFO:journalctl@ceph.osd.1.smithi086.stdout:Aug 06 17:50:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1.service: Consumed 2.055s CPU time. 2024-08-06T17:50:46.577 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sent signal SIGHUP to control process 59527 (bash) on client request. 2024-08-06T17:50:46.577 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Sending signal SIGHUP to process 59591 (podman) on client request. 2024-08-06T17:50:46.577 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Deactivated successfully. 2024-08-06T17:50:46.577 INFO:journalctl@ceph.osd.7.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7.service: Consumed 2.034s CPU time. 2024-08-06T17:50:46.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:46.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:46.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:46 smithi107 ceph-mon[32381]: osdmap e101: 12 total, 4 up, 12 in 2024-08-06T17:50:46.843 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:46.843 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:46.843 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:46 smithi107 podman[59545]: 2024-08-06 17:50:46.643995176 +0000 UTC m=+0.640106804 container remove 0af6ead3ffdddc409ab4e5f05cd5b219a9bb9458ffd9ea83ff3502dd909b91ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=main-93ef537, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-08-06T17:50:46.843 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Deactivated successfully. 2024-08-06T17:50:46.843 INFO:journalctl@ceph.osd.4.smithi107.stdout:Aug 06 17:50:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4.service: Consumed 2.197s CPU time. 2024-08-06T17:50:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:46 smithi086 ceph-mon[29229]: osdmap e101: 12 total, 4 up, 12 in 2024-08-06T17:50:46.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:46.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:46 smithi117 ceph-mon[32835]: osdmap e101: 12 total, 4 up, 12 in 2024-08-06T17:50:47.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:47.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:47.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:47.363 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sent signal SIGHUP to control process 72153 (bash) on client request. 2024-08-06T17:50:47.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 72154 (podman) on client request. 2024-08-06T17:50:47.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:47.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:47.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:47.505 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sent signal SIGHUP to main process 48951 (conmon) on client request. 2024-08-06T17:50:47.505 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 48953 (podman-init) on client request. 2024-08-06T17:50:47.505 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 48955 (ceph-osd) on client request. 2024-08-06T17:50:47.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:47.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:47 smithi086 ceph-mon[29229]: osdmap e102: 12 total, 4 up, 12 in 2024-08-06T17:50:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:47 smithi086 ceph-mon[29229]: pgmap v322: 33 pgs: 2 active+clean, 1 stale+active+clean, 12 undersized+peered, 18 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:47 smithi086 ceph-mon[29229]: osdmap e103: 12 total, 4 up, 12 in 2024-08-06T17:50:47.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sent signal SIGHUP to control process 72275 (bash) on client request. 2024-08-06T17:50:47.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 72276 (podman) on client request. 2024-08-06T17:50:47.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:47.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:47.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Deactivated successfully. 2024-08-06T17:50:47.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Unit process 43627 (podman-init) remains running after unit stopped. 2024-08-06T17:50:47.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Unit process 43629 (ceph-osd) remains running after unit stopped. 2024-08-06T17:50:47.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Consumed 2.284s CPU time. 2024-08-06T17:50:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:47 smithi117 ceph-mon[32835]: osdmap e102: 12 total, 4 up, 12 in 2024-08-06T17:50:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:47 smithi117 ceph-mon[32835]: pgmap v322: 33 pgs: 2 active+clean, 1 stale+active+clean, 12 undersized+peered, 18 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:47 smithi117 ceph-mon[32835]: osdmap e103: 12 total, 4 up, 12 in 2024-08-06T17:50:47.874 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sent signal SIGHUP to control process 59348 (bash) on client request. 2024-08-06T17:50:47.874 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 59394 (podman) on client request. 2024-08-06T17:50:47.874 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 48953 (podman-init) on client request. 2024-08-06T17:50:47.874 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:47 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 48955 (ceph-osd) on client request. 2024-08-06T17:50:47.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:47 smithi107 ceph-mon[32381]: osdmap e102: 12 total, 4 up, 12 in 2024-08-06T17:50:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:47 smithi107 ceph-mon[32381]: pgmap v322: 33 pgs: 2 active+clean, 1 stale+active+clean, 12 undersized+peered, 18 unknown; 577 KiB data, 326 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:47 smithi107 ceph-mon[32381]: osdmap e103: 12 total, 4 up, 12 in 2024-08-06T17:50:48.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:48.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:48.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:48.312 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:50:48.312 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:50:48.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T17:50:48.312 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:50:48.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:48.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:48.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:48.695 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sent signal SIGHUP to control process 72040 (bash) on client request. 2024-08-06T17:50:48.695 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 72346 (podman) on client request. 2024-08-06T17:50:48.695 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:48.695 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:48.699 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:50:48.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:48.847 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:48 smithi117 ceph-mon[32835]: osd.10 marked itself down and dead 2024-08-06T17:50:48.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:50:48.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:48 smithi107 ceph-mon[32381]: osd.10 marked itself down and dead 2024-08-06T17:50:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:50:48.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:48 smithi086 ceph-mon[29229]: osd.10 marked itself down and dead 2024-08-06T17:50:48.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:50:49.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:49.111 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 podman[59477]: 2024-08-06 17:50:49.052920525 +0000 UTC m=+1.318529386 container stop 4e6bf03a9b2ac2fbb7656d7bcd256aafa8783d61be1cee13a2223318f1e94858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-10, org.label-schema.vendor=CentOS, RELEASE=main-93ef537, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-08-06T17:50:49.112 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 podman[59477]: 2024-08-06 17:50:49.05335042 +0000 UTC m=+1.318959268 container died 4e6bf03a9b2ac2fbb7656d7bcd256aafa8783d61be1cee13a2223318f1e94858 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-08-06T17:50:49.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sent signal SIGHUP to control process 72410 (bash) on client request. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 72411 (podman) on client request. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Deactivated successfully. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Unit process 55037 (podman-init) remains running after unit stopped. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Unit process 55039 (ceph-osd) remains running after unit stopped. 2024-08-06T17:50:49.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Consumed 2.106s CPU time. 2024-08-06T17:50:49.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:49.373 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sent signal SIGHUP to control process 59424 (bash) on client request. 2024-08-06T17:50:49.373 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Sending signal SIGHUP to process 59477 (podman) on client request. 2024-08-06T17:50:49.373 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Deactivated successfully. 2024-08-06T17:50:49.373 INFO:journalctl@ceph.osd.10.smithi117.stdout:Aug 06 17:50:49 smithi117 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10.service: Consumed 1.566s CPU time. 2024-08-06T17:50:49.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:49.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:49.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:49.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:49.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: osdmap e104: 12 total, 3 up, 12 in 2024-08-06T17:50:49.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:49 smithi117 ceph-mon[32835]: pgmap v325: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: osdmap e104: 12 total, 3 up, 12 in 2024-08-06T17:50:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:49 smithi086 ceph-mon[29229]: pgmap v325: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:49.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:49.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: Health check failed: 1 host (4 osds) down (OSD_HOST_DOWN) 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2894099014' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: osdmap e104: 12 total, 3 up, 12 in 2024-08-06T17:50:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:49 smithi107 ceph-mon[32381]: pgmap v325: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:50.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:50.146 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:50:50.146 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:50:50.147 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:50:50.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:50.164 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:50:50.165 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:50:50.165 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:50:50.165 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:50:50.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:50:50.185 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:50:50.185 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:50:50.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:50:50.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:50:50.186 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:50:50.186 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:50:50.186 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:50:50.187 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:50.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:50.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:50.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:50.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:50.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:50.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:50.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:50 smithi086 ceph-mon[29229]: osdmap e105: 12 total, 3 up, 12 in 2024-08-06T17:50:50.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/519280483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:50:50.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:50.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:50.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:50 smithi107 ceph-mon[32381]: osdmap e105: 12 total, 3 up, 12 in 2024-08-06T17:50:50.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/519280483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:50:50.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:51.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:51.087 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:50 smithi117 ceph-mon[32835]: osdmap e105: 12 total, 3 up, 12 in 2024-08-06T17:50:51.087 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/519280483' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:50:51.087 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:51.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:51.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:51.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:51.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:50:51.494 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:51.495 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:51.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:51.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:51.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:51.660 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:50:51.661 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:50:51.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:51.800 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:51.800 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:51 smithi086 ceph-mon[29229]: osdmap e106: 12 total, 3 up, 12 in 2024-08-06T17:50:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:51 smithi086 ceph-mon[29229]: pgmap v328: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:51.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:52.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:52.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:52.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:52.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:51 smithi107 ceph-mon[32381]: osdmap e106: 12 total, 3 up, 12 in 2024-08-06T17:50:52.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:51 smithi107 ceph-mon[32381]: pgmap v328: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:52.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:52.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:50:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:51 smithi117 ceph-mon[32835]: osdmap e106: 12 total, 3 up, 12 in 2024-08-06T17:50:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:51 smithi117 ceph-mon[32835]: pgmap v328: 33 pgs: 4 stale+undersized+peered, 2 active+clean, 1 stale+active+clean, 13 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:52.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:52.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:52.383 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.139454018 +0000 UTC m=+0.449274242 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:50:52.383 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.270534913 +0000 UTC m=+0.580355169 container create fc29ed5dc6e8657e0d0fa84e8852419b926f97d58af901724573746d95140b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-deactivate, RELEASE=main-93ef537, ceph=True, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-08-06T17:50:52.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:52.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:52.593 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:52.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:52.709 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:52.710 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:52.710 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.506442612 +0000 UTC m=+0.816262837 container init fc29ed5dc6e8657e0d0fa84e8852419b926f97d58af901724573746d95140b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-08-06T17:50:52.710 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.50984967 +0000 UTC m=+0.819669893 container start fc29ed5dc6e8657e0d0fa84e8852419b926f97d58af901724573746d95140b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=main-93ef537, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-08-06T17:50:52.710 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.589650409 +0000 UTC m=+0.899470634 container attach fc29ed5dc6e8657e0d0fa84e8852419b926f97d58af901724573746d95140b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-deactivate, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=main-93ef537) 2024-08-06T17:50:52.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:52.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:53.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:53.116 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:50:52 smithi086 podman[72812]: 2024-08-06 17:50:52.840705789 +0000 UTC m=+1.150526004 container died fc29ed5dc6e8657e0d0fa84e8852419b926f97d58af901724573746d95140b7a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-deactivate, ceph=True, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-08-06T17:50:53.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:53.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:53.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:53.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:53.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:53.526 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:53.526 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:53.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:53.636 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:50:53.637 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:50:53.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:50:53.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:53.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:53.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo pkill -f 'journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service' 2024-08-06T17:50:54.035 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T17:50:54.036 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-08-06T17:50:54.036 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.3 2024-08-06T17:50:54.036 INFO:tasks.thrashosds.thrasher:remote for osd 3 is ubuntu@smithi086.front.sepia.ceph.com 2024-08-06T17:50:54.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell -n osd.3 --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-3 --journal-path /var/lib/ceph/osd/ceph-3/journal --op list-pgs 2024-08-06T17:50:54.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:54.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:54.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:54.406 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/osd.3/config 2024-08-06T17:50:54.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T17:50:54.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:54.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:54.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:54.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:54.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:54.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:54 smithi117 ceph-mon[32835]: pgmap v329: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:54.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:54 smithi086 ceph-mon[29229]: pgmap v329: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:54 smithi107 ceph-mon[32381]: pgmap v329: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:54.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:55.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:55.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:55.285 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:55.285 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:55.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:55.497 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:55.498 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:55.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:55.555 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:55.555 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:55.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:55.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:55.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:50:55.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:50:55.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:55 smithi086 ceph-mon[29229]: osdmap e107: 12 total, 3 up, 12 in 2024-08-06T17:50:55.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:55.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:50:55.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:50:55.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:55 smithi107 ceph-mon[32381]: osdmap e107: 12 total, 3 up, 12 in 2024-08-06T17:50:56.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:50:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:50:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:55 smithi117 ceph-mon[32835]: osdmap e107: 12 total, 3 up, 12 in 2024-08-06T17:50:56.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:56.203 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:56.224 INFO:teuthology.orchestra.run.smithi086.stderr:Creating an OSD daemon form without an OSD FSID value 2024-08-06T17:50:56.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:56.353 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:56.353 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:56.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:56.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:56.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:56.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:56.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:50:56.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:56.892 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:56 smithi117 ceph-mon[32835]: pgmap v331: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:56 smithi107 ceph-mon[32381]: pgmap v331: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:56.973 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:56 smithi086 ceph-mon[29229]: pgmap v331: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:56.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:50:56.974 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:50:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:50:57.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:50:57.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:50:57.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:57.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:57.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:57.569 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:57.569 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:57.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:57.704 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:50:57 smithi117 ceph-mon[32835]: pgmap v332: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:57.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:57.861 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:50:57 smithi086 ceph-mon[29229]: pgmap v332: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:57.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:50:57 smithi107 ceph-mon[32381]: pgmap v332: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:50:58.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:50:58.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:58.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:50:58.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:58.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:58.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:50:58.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:50:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:50:58.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:58.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:50:58.784 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:58.785 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:58.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:59.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:50:59.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:50:59.291 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:50:59.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:59.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:50:59.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:50:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:50:59.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:50:59.634 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:50:59.635 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:50:59.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:50:59.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:50:59.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:00.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:00.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:00.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:00.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:00.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:00.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:00.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:00.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:00.673 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:51:00.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:00.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T17:51:00.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:00.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:00 smithi086 ceph-mon[29229]: pgmap v333: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:00.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:00 smithi117 ceph-mon[32835]: pgmap v333: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:00.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:00 smithi107 ceph-mon[32381]: pgmap v333: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:01.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:01.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:01.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:01.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:01.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:01.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:01.754 INFO:tasks.thrashosds.thrasher:No PGs found for osd.3 2024-08-06T17:51:01.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:01 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:01 smithi107 ceph-mon[32381]: osdmap e108: 12 total, 3 up, 12 in 2024-08-06T17:51:02.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:02.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:02.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:02.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:01 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:02.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:02.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:01 smithi086 ceph-mon[29229]: osdmap e108: 12 total, 3 up, 12 in 2024-08-06T17:51:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:01 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3443303121' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:01 smithi117 ceph-mon[32835]: osdmap e108: 12 total, 3 up, 12 in 2024-08-06T17:51:02.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:02.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:02.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:02.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:02.638 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:02.649 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:02.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:02.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:02.668 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:02.668 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:02.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:02.669 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:02.669 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:02.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:02.687 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:02.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:02.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:02.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:02 smithi117 ceph-mon[32835]: pgmap v335: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:02.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/320174034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:02 smithi086 ceph-mon[29229]: pgmap v335: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/320174034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:02.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:02 smithi107 ceph-mon[32381]: pgmap v335: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/320174034' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:03.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:03.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:03.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:03.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:51:03.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:03.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:03.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:03.932 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:03.933 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:03.933 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:03.933 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:03.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.958 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:03.959 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:04.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:04.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:04.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:04.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:04.456 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:04.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:04.679 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:04.679 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:04.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:04.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:04.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:04 smithi086 ceph-mon[29229]: pgmap v336: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:04.963 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:04.963 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:04 smithi107 ceph-mon[32381]: pgmap v336: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:04.972 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:04.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:05.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:05.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:04 smithi117 ceph-mon[32835]: pgmap v336: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:05.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:05.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:05.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:05.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:05.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:05.552 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:05.552 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:05.552 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:05.553 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:05.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:05.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:05.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:05 smithi086 ceph-mon[29229]: pgmap v337: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:05.814 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:05.814 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:05.870 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:05.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T17:51:05.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:05.949 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:05 smithi107 ceph-mon[32381]: pgmap v337: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:05.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:51:06.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:06.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:06.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:05 smithi117 ceph-mon[32835]: pgmap v337: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:06.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:06.308 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:06.318 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:06.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:06.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:06.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:06.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:06.755 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [3] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-06T17:51:06.755 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-06T17:51:06.756 INFO:tasks.thrashosds.thrasher:Reviving osd 3 2024-08-06T17:51:06.756 INFO:tasks.cephadm.osd.3:Restarting osd.3 (starting--it wasn't running)... 2024-08-06T17:51:06.756 DEBUG:teuthology.orchestra.run.smithi086:osd.3> sudo journalctl -f -n 0 -u ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service 2024-08-06T17:51:06.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl start ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:06.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:06.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:06.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:06.996 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:06 smithi086 systemd[1]: Starting Ceph osd.3 for fd7e697e-541a-11ef-bccc-c7b262605968... 2024-08-06T17:51:06.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:07.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:07.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:07.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:07.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:07.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:07.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:07.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:07.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:07.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:08.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:08.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:08.101 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:08.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:08.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:08.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:08.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:08.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:08.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:08.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:08 smithi086 ceph-mon[29229]: pgmap v338: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:08 smithi117 ceph-mon[32835]: pgmap v338: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:08.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:08 smithi107 ceph-mon[32381]: pgmap v338: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:08.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:08.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:09.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:09.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:09.223 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:09.223 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:09.223 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:09 smithi086 podman[74805]: 2024-08-06 17:51:09.022665023 +0000 UTC m=+0.446879398 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:51:09.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:09.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:09.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:09.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:09.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:09.674 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:51:09.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:09.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:09.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:09.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:09.851 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:09 smithi086 podman[74805]: 2024-08-06 17:51:09.786218254 +0000 UTC m=+1.210432616 container create 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0) 2024-08-06T17:51:09.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:10.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:10.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:10.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:10.395 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 podman[74805]: 2024-08-06 17:51:10.264681382 +0000 UTC m=+1.688895742 container init 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-08-06T17:51:10.395 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 podman[74805]: 2024-08-06 17:51:10.268034517 +0000 UTC m=+1.692248885 container start 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , RELEASE=main-93ef537, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True) 2024-08-06T17:51:10.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:10.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-06T17:51:10.671 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate[74950]: --> ceph-volume raw activate successful for osd ID: 3 2024-08-06T17:51:10.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:10.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:10.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:10 smithi117 ceph-mon[32835]: pgmap v339: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:10.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:10 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:10.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:10.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:10 smithi117 ceph-mon[32835]: osdmap e109: 12 total, 3 up, 12 in 2024-08-06T17:51:10.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:10.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:10 smithi107 ceph-mon[32381]: pgmap v339: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:10 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:10 smithi107 ceph-mon[32381]: osdmap e109: 12 total, 3 up, 12 in 2024-08-06T17:51:11.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:11.058 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:11.058 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:11.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-mon[29229]: pgmap v339: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:11.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:11.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2378137048' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:11.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:10 smithi086 ceph-mon[29229]: osdmap e109: 12 total, 3 up, 12 in 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 podman[74805]: 2024-08-06 17:51:10.792734736 +0000 UTC m=+2.216949099 container attach 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, RELEASE=main-93ef537, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-3 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-08-06T17:51:11.059 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-08-06T17:51:11.060 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-3/block 2024-08-06T17:51:11.060 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-08-06T17:51:11.060 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 bash[74805]: --> ceph-volume raw activate successful for osd ID: 3 2024-08-06T17:51:11.060 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:10 smithi086 podman[75088]: 2024-08-06 17:51:10.793540969 +0000 UTC m=+0.141048558 container died 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=main-93ef537, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-08-06T17:51:11.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:11.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:11.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:11.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:11.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:11.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:11.581 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:11 smithi086 podman[75088]: 2024-08-06 17:51:11.354483574 +0000 UTC m=+0.701991153 container remove 2e5cb5a107efe60513bfd406c1a2a0ef92e7c5a91e7d22409e566ac041ce0bb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3-activate, org.label-schema.schema-version=1.0, RELEASE=main-93ef537, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-08-06T17:51:11.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:11.816 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:11.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:11.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:11.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:11.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:12.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:12.160 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:11 smithi086 podman[75230]: 2024-08-06 17:51:11.814453115 +0000 UTC m=+0.077959779 image pull c36b274456db3b2120320e088a5598d1697e123fbf67e981d4d5c41be5535f1c quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60 2024-08-06T17:51:12.161 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 podman[75230]: 2024-08-06 17:51:12.101141572 +0000 UTC m=+0.364648228 container create be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, ceph=True, GIT_BRANCH=HEAD, RELEASE=main-93ef537, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-08-06T17:51:12.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:12.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:12.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:12 smithi086 ceph-mon[29229]: osdmap e110: 12 total, 3 up, 12 in 2024-08-06T17:51:12.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:12 smithi086 ceph-mon[29229]: pgmap v342: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:12.420 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 podman[75230]: 2024-08-06 17:51:12.395108962 +0000 UTC m=+0.658615621 container init be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=main-93ef537, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-08-06T17:51:12.420 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 podman[75230]: 2024-08-06 17:51:12.398432449 +0000 UTC m=+0.661939106 container start be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, RELEASE=main-93ef537, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-06T17:51:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:12 smithi107 ceph-mon[32381]: osdmap e110: 12 total, 3 up, 12 in 2024-08-06T17:51:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:12 smithi107 ceph-mon[32381]: pgmap v342: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:12.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T17:51:12.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:12 smithi117 ceph-mon[32835]: osdmap e110: 12 total, 3 up, 12 in 2024-08-06T17:51:12.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:12 smithi117 ceph-mon[32835]: pgmap v342: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:12.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:12.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:51:12.726 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 ceph-osd[75278]: -- 172.21.15.86:0/3012152742 <== mon.0 v2:172.21.15.86:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ffbd1ffa40 con 0x55ffbdfc8000 2024-08-06T17:51:12.726 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 bash[75230]: be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 2024-08-06T17:51:12.726 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 systemd[1]: Started Ceph osd.3 for fd7e697e-541a-11ef-bccc-c7b262605968. 2024-08-06T17:51:12.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:12.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:12.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T17:51:12.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:13.017 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:12 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[75258]: 2024-08-06T17:51:12.951+0000 7f580f552640 -1 Falling back to public interface 2024-08-06T17:51:13.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:13.155 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:13.156 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:13.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:13.187 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:13.187 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:13.188 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:13.209 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:13.209 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:13.210 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:13.210 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:13.210 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:13.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:13.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:13.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:13.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:13.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:13.343 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3592110495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:13.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3592110495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:13.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3592110495' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:13.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:13.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:13.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:13.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:14.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:14.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:14.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:14.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:14.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:14.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:14 smithi117 ceph-mon[32835]: pgmap v343: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:14.439 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:14.439 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:14.439 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:14 smithi086 ceph-mon[29229]: pgmap v343: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:14.444 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:14.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:14.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:14.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:14 smithi107 ceph-mon[32381]: pgmap v343: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:14.471 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:14.471 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:14.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:14.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:14.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:14.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:15.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:15.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:15.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:15.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:15.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:15.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:15.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:15.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:15.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:15.968 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:15 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[75258]: 2024-08-06T17:51:15.684+0000 7f580f552640 -1 osd.3 98 log_to_monitors true 2024-08-06T17:51:16.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:16.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:16.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:16.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:16.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:16.331 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3[75258]: 2024-08-06T17:51:16.178+0000 7f58032f4640 -1 osd.3 98 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-08-06T17:51:16.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:16.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:51:16.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:16.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:16.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:16.867 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:16 smithi117 ceph-mon[32835]: pgmap v344: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:16.867 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:16 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:51:16.867 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:16 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:51:16.868 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:16 smithi117 ceph-mon[32835]: osdmap e111: 12 total, 3 up, 12 in 2024-08-06T17:51:16.868 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:16 smithi117 ceph-mon[32835]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:51:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:16 smithi107 ceph-mon[32381]: pgmap v344: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:16 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:51:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:16 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:51:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:16 smithi107 ceph-mon[32381]: osdmap e111: 12 total, 3 up, 12 in 2024-08-06T17:51:16.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:16 smithi107 ceph-mon[32381]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:51:16.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:17.021 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:17.021 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:17.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-mon[29229]: pgmap v344: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:17.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]} : dispatch 2024-08-06T17:51:17.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-08-06T17:51:17.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-mon[29229]: osdmap e111: 12 total, 3 up, 12 in 2024-08-06T17:51:17.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:16 smithi086 ceph-mon[29229]: from='osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870]' entity='osd.3' cmd={"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi086", "root=default"]} : dispatch 2024-08-06T17:51:17.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:51:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:51:17.238 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T17:51:17.238 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:51:17.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:17.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:17.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:17.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:17.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:17.566 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:17.566 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:17.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:17.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:17.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:17.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:17.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:17 smithi117 ceph-mon[32835]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:17.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:17 smithi117 ceph-mon[32835]: osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870] boot 2024-08-06T17:51:17.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:17 smithi117 ceph-mon[32835]: osdmap e112: 12 total, 4 up, 12 in 2024-08-06T17:51:17.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:17 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:51:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:17 smithi086 ceph-mon[29229]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:17 smithi086 ceph-mon[29229]: osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870] boot 2024-08-06T17:51:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:17 smithi086 ceph-mon[29229]: osdmap e112: 12 total, 4 up, 12 in 2024-08-06T17:51:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:17 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:51:17.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:17.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:17.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:17 smithi107 ceph-mon[32381]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:17 smithi107 ceph-mon[32381]: osd.3 [v2:172.21.15.86:6810/1831259870,v1:172.21.15.86:6811/1831259870] boot 2024-08-06T17:51:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:17 smithi107 ceph-mon[32381]: osdmap e112: 12 total, 4 up, 12 in 2024-08-06T17:51:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:17 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "osd metadata", "id": 3} : dispatch 2024-08-06T17:51:18.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:18.168 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:51:18.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:18.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:18.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:18.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:18.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:18.586 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:18.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:18.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:18.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:18.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:18.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:18.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:18.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:18 smithi107 ceph-mon[32381]: pgmap v347: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:18.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:18.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:18 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:18.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:18.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:18 smithi107 ceph-mon[32381]: osdmap e113: 12 total, 4 up, 12 in 2024-08-06T17:51:18.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:18.899 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:18.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:18 smithi117 ceph-mon[32835]: pgmap v347: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:18.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:18.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:18 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:18.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:18.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:18 smithi117 ceph-mon[32835]: osdmap e113: 12 total, 4 up, 12 in 2024-08-06T17:51:19.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:19.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:18 smithi086 ceph-mon[29229]: pgmap v347: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:19.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:19.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:18 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:19.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4156698349' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:19.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:18 smithi086 ceph-mon[29229]: osdmap e113: 12 total, 4 up, 12 in 2024-08-06T17:51:19.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:19.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:19.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:19.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:19.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:19.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:19.580 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:19.580 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:19.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:19.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:19 smithi086 ceph-mon[29229]: osdmap e114: 12 total, 4 up, 12 in 2024-08-06T17:51:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:19 smithi117 ceph-mon[32835]: osdmap e114: 12 total, 4 up, 12 in 2024-08-06T17:51:19.925 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:19.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T17:51:19.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:19 smithi107 ceph-mon[32381]: osdmap e114: 12 total, 4 up, 12 in 2024-08-06T17:51:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:20.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:20.197 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sent signal SIGHUP to main process 75258 (conmon) on client request. 2024-08-06T17:51:20.197 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 75260 (podman-init) on client request. 2024-08-06T17:51:20.198 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 75278 (ceph-osd) on client request. 2024-08-06T17:51:20.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:20.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:20.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:20.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:20.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:20.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:20.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:20.799 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:20 smithi107 ceph-mon[32381]: pgmap v350: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:20 smithi086 ceph-mon[29229]: pgmap v350: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:20.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:20.987 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:21.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:20 smithi117 ceph-mon[32835]: pgmap v350: 33 pgs: 4 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 13 unknown; 577 KiB data, 330 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:21.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:21.214 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:21.214 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:21.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:21.366 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sent signal SIGHUP to control process 76654 (bash) on client request. 2024-08-06T17:51:21.366 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 76655 (podman) on client request. 2024-08-06T17:51:21.366 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 75260 (podman-init) on client request. 2024-08-06T17:51:21.366 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 75278 (ceph-osd) on client request. 2024-08-06T17:51:21.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:21.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:21.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:21.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:21.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:22.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:22.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:22 smithi117 ceph-mon[32835]: osdmap e115: 12 total, 4 up, 12 in 2024-08-06T17:51:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:22 smithi117 ceph-mon[32835]: pgmap v352: 33 pgs: 4 active+clean, 4 stale+undersized+peered, 1 stale+active+clean, 12 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:22.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:22.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:22 smithi086 ceph-mon[29229]: osdmap e115: 12 total, 4 up, 12 in 2024-08-06T17:51:22.419 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:22 smithi086 ceph-mon[29229]: pgmap v352: 33 pgs: 4 active+clean, 4 stale+undersized+peered, 1 stale+active+clean, 12 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:22 smithi107 ceph-mon[32381]: osdmap e115: 12 total, 4 up, 12 in 2024-08-06T17:51:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:22 smithi107 ceph-mon[32381]: pgmap v352: 33 pgs: 4 active+clean, 4 stale+undersized+peered, 1 stale+active+clean, 12 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:22.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:22.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:51:22.615 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:22.616 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:22.616 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:22.632 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:22.649 INFO:teuthology.orchestra.run.smithi086.stdout:osd.3: {} 2024-08-06T17:51:22.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.3: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-08-06T17:51:22.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:22.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:22.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:22.666 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:22.667 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:22.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:22.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:22.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:22.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:22.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:22.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:22.819 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:22.819 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:22.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:23.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:23.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:23.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:23.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:23 smithi086 ceph-mon[29229]: osd.3 marked itself down and dead 2024-08-06T17:51:23.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1235747387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:23.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:23 smithi117 ceph-mon[32835]: osd.3 marked itself down and dead 2024-08-06T17:51:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1235747387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:23 smithi107 ceph-mon[32381]: osd.3 marked itself down and dead 2024-08-06T17:51:23.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1235747387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:23.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:23.466 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:23.466 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:23.466 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:23.466 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:23.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:23.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:23.700 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:23.701 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:23.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:23.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:23.939 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 podman[76817]: 2024-08-06 17:51:23.723261108 +0000 UTC m=+2.270683609 container stop be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, RELEASE=main-93ef537, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-08-06T17:51:23.940 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 podman[76817]: 2024-08-06 17:51:23.723719857 +0000 UTC m=+2.271142355 container died be39d4ba43744a6e0a383bc406854c6492591d94b2d75ede537c9c863c4ac5b1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:f79be09cb4cad27794716a0f93f517b237f9a2a419d1204e32d93e6703d63e60, name=ceph-fd7e697e-541a-11ef-bccc-c7b262605968-osd-3, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, RELEASE=main-93ef537, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=c5aaba5e3282b30e4782f2b5d6e4e362e22dfcb7, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-08-06T17:51:23.940 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sent signal SIGHUP to control process 76729 (bash) on client request. 2024-08-06T17:51:23.940 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Sending signal SIGHUP to process 76817 (podman) on client request. 2024-08-06T17:51:23.940 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Deactivated successfully. 2024-08-06T17:51:23.940 INFO:journalctl@ceph.osd.3.smithi086.stdout:Aug 06 17:51:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3.service: Consumed 3.585s CPU time. 2024-08-06T17:51:24.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:24.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:24.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:24.210 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:24 smithi086 ceph-mon[29229]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:51:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:24 smithi086 ceph-mon[29229]: osdmap e116: 12 total, 3 up, 12 in 2024-08-06T17:51:24.211 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:24 smithi086 ceph-mon[29229]: pgmap v354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 3 stale+active+clean, 11 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:24.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:24.298 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:24.312 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:51:24.312 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:51:24.312 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T17:51:24.312 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:51:24.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:24 smithi107 ceph-mon[32381]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:51:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:24 smithi107 ceph-mon[32381]: osdmap e116: 12 total, 3 up, 12 in 2024-08-06T17:51:24.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:24 smithi107 ceph-mon[32381]: pgmap v354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 3 stale+active+clean, 11 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:24.498 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-08-06T17:51:24.498 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-08-06T17:51:24.499 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11] 2024-08-06T17:51:24.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd out 3 2024-08-06T17:51:24.520 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:24.520 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:24.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:24.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:24.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:24 smithi117 ceph-mon[32835]: Health check update: 9 osds down (OSD_DOWN) 2024-08-06T17:51:24.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:24 smithi117 ceph-mon[32835]: osdmap e116: 12 total, 3 up, 12 in 2024-08-06T17:51:24.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:24 smithi117 ceph-mon[32835]: pgmap v354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 3 stale+active+clean, 11 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:24.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:24.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:24.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:25.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:25.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:25.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:25 smithi117 ceph-mon[32835]: osdmap e117: 12 total, 3 up, 12 in 2024-08-06T17:51:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:25 smithi107 ceph-mon[32381]: osdmap e117: 12 total, 3 up, 12 in 2024-08-06T17:51:25.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:25.488 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:25 smithi086 ceph-mon[29229]: osdmap e117: 12 total, 3 up, 12 in 2024-08-06T17:51:25.489 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:25.489 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:25.489 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:25.489 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:25.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:25.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:51:25.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:25.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:26.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:26.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:26.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T17:51:26.184 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:26.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:26 smithi117 ceph-mon[32835]: osdmap e118: 12 total, 3 up, 12 in 2024-08-06T17:51:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:26 smithi117 ceph-mon[32835]: pgmap v357: 33 pgs: 5 stale+undersized+peered, 3 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:26 smithi117 ceph-mon[32835]: osdmap e119: 12 total, 3 up, 12 in 2024-08-06T17:51:26.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:26 smithi086 ceph-mon[29229]: osdmap e118: 12 total, 3 up, 12 in 2024-08-06T17:51:26.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:26 smithi086 ceph-mon[29229]: pgmap v357: 33 pgs: 5 stale+undersized+peered, 3 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:26.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:26 smithi086 ceph-mon[29229]: osdmap e119: 12 total, 3 up, 12 in 2024-08-06T17:51:26.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:26.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:26 smithi107 ceph-mon[32381]: osdmap e118: 12 total, 3 up, 12 in 2024-08-06T17:51:26.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:26 smithi107 ceph-mon[32381]: pgmap v357: 33 pgs: 5 stale+undersized+peered, 3 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:26.451 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:26 smithi107 ceph-mon[32381]: osdmap e119: 12 total, 3 up, 12 in 2024-08-06T17:51:26.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:26.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:26.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:26.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:27.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:27.204 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:51:27.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:27.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:27.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:27.391 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:27.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:27.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:27.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:27.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:27.791 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:27.791 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:27.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:28.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:28.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:28.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:28.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:28.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:28.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:28.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:28 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:51:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:28 smithi117 ceph-mon[32835]: osdmap e120: 12 total, 3 up, 12 in 2024-08-06T17:51:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:28 smithi117 ceph-mon[32835]: pgmap v360: 33 pgs: 5 stale+undersized+peered, 2 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:28.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:28 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:51:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:28 smithi107 ceph-mon[32381]: osdmap e120: 12 total, 3 up, 12 in 2024-08-06T17:51:28.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:28 smithi107 ceph-mon[32381]: pgmap v360: 33 pgs: 5 stale+undersized+peered, 2 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:28.512 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:28 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:28.512 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/416920880' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:51:28.512 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:28 smithi086 ceph-mon[29229]: osdmap e120: 12 total, 3 up, 12 in 2024-08-06T17:51:28.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:28 smithi086 ceph-mon[29229]: pgmap v360: 33 pgs: 5 stale+undersized+peered, 2 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:28.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:28.694 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:28.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:28.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:28.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:29.073 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:29.073 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:29.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:29.217 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-fd7e697e-541a-11ef-bccc-c7b262605968-mon-a[29225]: 2024-08-06T17:51:29.073+0000 7f8fdf88e640 -1 mon.a@0(leader).osd e120 definitely_dead 0 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-06T17:51:29.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:29 smithi086 ceph-mon[29229]: Client client.admin marked osd.3 out, after it was down for 5 seconds 2024-08-06T17:51:29.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:29.374 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:29.375 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-06T17:51:29.375 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:29 smithi107 ceph-mon[32381]: Client client.admin marked osd.3 out, after it was down for 5 seconds 2024-08-06T17:51:29.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:29.561 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:29.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd={"prefix": "osd out", "ids": ["3"]} : dispatch 2024-08-06T17:51:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:29 smithi117 ceph-mon[32835]: Client client.admin marked osd.3 out, after it was down for 5 seconds 2024-08-06T17:51:29.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:29.813 INFO:teuthology.orchestra.run.smithi086.stderr:marked out osd.3. 2024-08-06T17:51:29.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:30.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:30.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:30.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:30.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:30 smithi086 ceph-mon[29229]: pgmap v361: 33 pgs: 5 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:30.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:30 smithi086 ceph-mon[29229]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:30.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-06T17:51:30.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:30 smithi086 ceph-mon[29229]: osdmap e121: 12 total, 3 up, 11 in 2024-08-06T17:51:30.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:51:30.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:30.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:30.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:30 smithi107 ceph-mon[32381]: pgmap v361: 33 pgs: 5 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:30.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:30 smithi107 ceph-mon[32381]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:30.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-06T17:51:30.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:30 smithi107 ceph-mon[32381]: osdmap e121: 12 total, 3 up, 11 in 2024-08-06T17:51:30.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:30.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:30 smithi117 ceph-mon[32835]: pgmap v361: 33 pgs: 5 stale+undersized+peered, 1 stale+active+clean, 15 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 1.0 TiB / 1.0 TiB avail 2024-08-06T17:51:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:30 smithi117 ceph-mon[32835]: Health check update: 8 osds down (OSD_DOWN) 2024-08-06T17:51:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/828781118' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-08-06T17:51:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:30 smithi117 ceph-mon[32835]: osdmap e121: 12 total, 3 up, 11 in 2024-08-06T17:51:30.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:30.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:30.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:30.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:31.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:31.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:31.178 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:51:31.178 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:51:31.178 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T17:51:31.178 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T17:51:31.178 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:51:31.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:31.290 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:31.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:31.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:31.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:31.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:31.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:31.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:31.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:31 smithi117 ceph-mon[32835]: osdmap e122: 12 total, 3 up, 11 in 2024-08-06T17:51:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:31 smithi117 ceph-mon[32835]: osdmap e123: 12 total, 3 up, 11 in 2024-08-06T17:51:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:31 smithi117 ceph-mon[32835]: pgmap v365: 33 pgs: 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:31.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:31.993 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:31 smithi107 ceph-mon[32381]: osdmap e122: 12 total, 3 up, 11 in 2024-08-06T17:51:31.993 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:31 smithi107 ceph-mon[32381]: osdmap e123: 12 total, 3 up, 11 in 2024-08-06T17:51:31.993 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:31 smithi107 ceph-mon[32381]: pgmap v365: 33 pgs: 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:32.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:31 smithi086 ceph-mon[29229]: osdmap e122: 12 total, 3 up, 11 in 2024-08-06T17:51:32.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:31 smithi086 ceph-mon[29229]: osdmap e123: 12 total, 3 up, 11 in 2024-08-06T17:51:32.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:31 smithi086 ceph-mon[29229]: pgmap v365: 33 pgs: 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 14 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:32.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:32.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:32.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:32.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T17:51:32.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:32.512 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:32.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:32.771 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:32.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:32.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:32.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:33.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:33.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:33.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:33.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:33.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:33.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:33.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:33.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:33.736 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3352909704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:33.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:33.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3352909704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:33.870 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:33.870 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:33.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:34.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3352909704' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:34.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:34.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:34.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:34.020 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:34.021 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:34.021 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:34.021 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:34.021 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:34.021 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:34.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:34.237 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:34.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:34.273 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:34.273 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:34.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:34.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:34.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:34.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:34.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:34.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:34.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:34.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:34 smithi086 ceph-mon[29229]: pgmap v366: 33 pgs: 1 active+clean, 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:34.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:34.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:34.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:34 smithi107 ceph-mon[32381]: pgmap v366: 33 pgs: 1 active+clean, 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:35.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:34 smithi117 ceph-mon[32835]: pgmap v366: 33 pgs: 1 active+clean, 1 peering, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:35.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:35.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:35.380 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:35.380 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:35.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:35.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:35.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:35.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:35.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:35.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:35.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:36.095 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 4, 5, 6, 7, 8, 9, 10, 11, 3] 2024-08-06T17:51:36.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 0 1 2024-08-06T17:51:36.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:36.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:36.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:36.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:36.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:36.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:36.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:36.676 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:51:36.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:36.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:36.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:36 smithi086 ceph-mon[29229]: pgmap v367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:36.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:36 smithi107 ceph-mon[32381]: pgmap v367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:37.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:36 smithi117 ceph-mon[32835]: pgmap v367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:37.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:37.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:37.372 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:37.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:51:37.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:37.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:37.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:37.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:37.596 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:37.597 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:37.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:37.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:37 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:37 smithi117 ceph-mon[32835]: osdmap e124: 12 total, 3 up, 11 in 2024-08-06T17:51:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:37 smithi117 ceph-mon[32835]: pgmap v369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:37 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:37 smithi107 ceph-mon[32381]: osdmap e124: 12 total, 3 up, 11 in 2024-08-06T17:51:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:37 smithi107 ceph-mon[32381]: pgmap v369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:37.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:37 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:37.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1458777440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:37.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:37 smithi086 ceph-mon[29229]: osdmap e124: 12 total, 3 up, 11 in 2024-08-06T17:51:37.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:37 smithi086 ceph-mon[29229]: pgmap v369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:37.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:38.002 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:51:38.002 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:51:38.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:38.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:38.289 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:38.289 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:38.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:38.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:38.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:38.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:38.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:39.061 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:39.061 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:39.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:39.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T17:51:39.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:39.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:39.350 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:39.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:39.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:39.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:51:39.418 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:39.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:39.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:39.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:39.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:39.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:39.810 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:39.810 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:39.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:40.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:40.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:40.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:40.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:40.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:40.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:40.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:40 smithi107 ceph-mon[32381]: pgmap v370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:40.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:40.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:40 smithi086 ceph-mon[29229]: pgmap v370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:40.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:40.987 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:41.009 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:41.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:41.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:40 smithi117 ceph-mon[32835]: pgmap v370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:41.168 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:41.168 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:41.168 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.0 to 1 (10000) 2024-08-06T17:51:41.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:41.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:41.416 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:41.417 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:41.417 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:41.434 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:41.434 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:41.434 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:41.434 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:41.435 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:41.435 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:41.435 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:41.451 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:41.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:41.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:41.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:41.678 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:41.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T17:51:41.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:41.762 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-06T17:51:41.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-06T17:51:41.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:41 smithi086 ceph-mon[29229]: osdmap e125: 12 total, 3 up, 11 in 2024-08-06T17:51:41.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2216679620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:41.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:41.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:41.930 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:41.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-06T17:51:41.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-06T17:51:41.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:41 smithi107 ceph-mon[32381]: osdmap e125: 12 total, 3 up, 11 in 2024-08-06T17:51:41.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2216679620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:41.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:42.027 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd={"prefix": "osd reweight", "id": 0, "weight": 1.0} : dispatch 2024-08-06T17:51:42.027 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2750450821' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2024-08-06T17:51:42.027 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:41 smithi117 ceph-mon[32835]: osdmap e125: 12 total, 3 up, 11 in 2024-08-06T17:51:42.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2216679620' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:42.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:42.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:42.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:42.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:42.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:42.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:42.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:42.590 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:42.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:42.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:42 smithi086 ceph-mon[29229]: pgmap v372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:42 smithi117 ceph-mon[32835]: pgmap v372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:42.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:42.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:42 smithi107 ceph-mon[32381]: pgmap v372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:43.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:43.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:43.288 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:43.289 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:43.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:43.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:43.583 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:43.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:43.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 1 1 2024-08-06T17:51:43.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:43.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:43 smithi086 ceph-mon[29229]: pgmap v373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:43.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:43.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:43.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:44.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:44.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:43 smithi107 ceph-mon[32381]: pgmap v373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:43 smithi117 ceph-mon[32835]: pgmap v373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:44.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:44.205 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:44.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T17:51:44.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:44.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:44.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:44.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:44.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:44.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:44.451 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:44.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:44.573 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:44.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:51:44.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:44.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:44.825 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:51:44.828 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:44.828 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/362278936' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:44 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:44.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:45.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:45.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/362278936' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:45.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:44 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:45.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:45.210 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/362278936' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:45.210 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:44 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:51:45.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:45.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:45.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:45.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:45.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:45.994 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:46.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:46.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:45 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:46.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:45 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:46.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:45 smithi086 ceph-mon[29229]: osdmap e126: 12 total, 3 up, 11 in 2024-08-06T17:51:46.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:45 smithi086 ceph-mon[29229]: pgmap v375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:45 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:45 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:46.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:45 smithi117 ceph-mon[32835]: osdmap e126: 12 total, 3 up, 11 in 2024-08-06T17:51:46.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:45 smithi117 ceph-mon[32835]: pgmap v375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:46.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:46.182 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:46.182 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:46.182 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:45 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:51:46.182 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:45 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:51:46.182 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:45 smithi107 ceph-mon[32381]: osdmap e126: 12 total, 3 up, 11 in 2024-08-06T17:51:46.182 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:45 smithi107 ceph-mon[32381]: pgmap v375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 303 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:46.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:46.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:46.443 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:46.443 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:46.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:46.622 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:46.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T17:51:46.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:46.714 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:46.714 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:46.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:46.874 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:46.934 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:46.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:47.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:46 smithi086 ceph-mon[29229]: Health check failed: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-08-06T17:51:47.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:46 smithi086 ceph-mon[29229]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:47.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:46 smithi086 ceph-mon[29229]: osdmap e127: 12 total, 3 up, 11 in 2024-08-06T17:51:47.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:47.106 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:47.106 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:47.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:46 smithi107 ceph-mon[32381]: Health check failed: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-08-06T17:51:47.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:46 smithi107 ceph-mon[32381]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:47.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:46 smithi107 ceph-mon[32381]: osdmap e127: 12 total, 3 up, 11 in 2024-08-06T17:51:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:46 smithi117 ceph-mon[32835]: Health check failed: Reduced data availability: 24 pgs inactive (PG_AVAILABILITY) 2024-08-06T17:51:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:46 smithi117 ceph-mon[32835]: Health check failed: Degraded data redundancy: 8 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:46 smithi117 ceph-mon[32835]: osdmap e127: 12 total, 3 up, 11 in 2024-08-06T17:51:47.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:47.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:47.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:47.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:47.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:47.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:47.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:47 smithi086 ceph-mon[29229]: pgmap v377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:48.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:48.040 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:47 smithi107 ceph-mon[32381]: pgmap v377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:47 smithi117 ceph-mon[32835]: pgmap v377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:48.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:48.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:48.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:48.491 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:48.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:51:48.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:48.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:48.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:48.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:48.965 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:48.966 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:48.966 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:48.983 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:48.984 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:48.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:48.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:48.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:48.985 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.1 to 1 (10000) 2024-08-06T17:51:49.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:49.013 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:49.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:49.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:49.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-06T17:51:49.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2059439772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:49.110 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:49.110 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:49.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:49.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T17:51:49.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-06T17:51:49.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2059439772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:49.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:49.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd={"prefix": "osd reweight", "id": 1, "weight": 1.0} : dispatch 2024-08-06T17:51:49.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2059439772' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:49.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:49.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:49.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:49.529 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:49.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:49.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:49.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:49.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:49.834 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:49.835 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:49.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:50.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: osdmap e128: 12 total, 3 up, 11 in 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:50 smithi086 ceph-mon[29229]: pgmap v379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:50.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: osdmap e128: 12 total, 3 up, 11 in 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:50 smithi107 ceph-mon[32381]: pgmap v379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:50.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2083008298' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-08-06T17:51:50.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: osdmap e128: 12 total, 3 up, 11 in 2024-08-06T17:51:50.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:50.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:50 smithi117 ceph-mon[32835]: pgmap v379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:50.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:50.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:50.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:50.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:50.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:50.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:50.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:51.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 2 1 2024-08-06T17:51:51.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:51.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:51 smithi117 ceph-mon[32835]: osdmap e129: 12 total, 3 up, 11 in 2024-08-06T17:51:51.204 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:51.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:51 smithi107 ceph-mon[32381]: osdmap e129: 12 total, 3 up, 11 in 2024-08-06T17:51:51.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:51.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:51 smithi086 ceph-mon[29229]: osdmap e129: 12 total, 3 up, 11 in 2024-08-06T17:51:51.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:51.452 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:51.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:51.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:51.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:51.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:51:51.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:51.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T17:51:51.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:51.911 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:51.912 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:52.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:52.109 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:52.109 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:52.110 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:52 smithi107 ceph-mon[32381]: Health check update: Reduced data availability: 29 pgs inactive, 5 pgs stale (PG_AVAILABILITY) 2024-08-06T17:51:52.110 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:52 smithi107 ceph-mon[32381]: Health check update: Degraded data redundancy: 13 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:52.110 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:52 smithi107 ceph-mon[32381]: pgmap v381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:52.126 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:52.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:52.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:52 smithi086 ceph-mon[29229]: Health check update: Reduced data availability: 29 pgs inactive, 5 pgs stale (PG_AVAILABILITY) 2024-08-06T17:51:52.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:52 smithi086 ceph-mon[29229]: Health check update: Degraded data redundancy: 13 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:52.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:52 smithi086 ceph-mon[29229]: pgmap v381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:52.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:52 smithi117 ceph-mon[32835]: Health check update: Reduced data availability: 29 pgs inactive, 5 pgs stale (PG_AVAILABILITY) 2024-08-06T17:51:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:52 smithi117 ceph-mon[32835]: Health check update: Degraded data redundancy: 13 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:52 smithi117 ceph-mon[32835]: pgmap v381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:52.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:52.556 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:52.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:52.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:52.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:52.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:52.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:52.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:53.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:53.184 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:51:53.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:53.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:53.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:53.457 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:53.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:53.520 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:53.520 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:53.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:51:53.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:53.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:53.767 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:51:53.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:53.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:53.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:54.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:54.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:54.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:54.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:54.450 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:54.451 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:54.451 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: osdmap e130: 12 total, 3 up, 11 in 2024-08-06T17:51:54.451 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: pgmap v383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:54.451 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:51:54.451 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:54 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:51:54.452 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:54.452 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: osdmap e130: 12 total, 3 up, 11 in 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: pgmap v383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:51:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:54 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:51:54.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:51:54.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/772155737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:51:54.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: osdmap e130: 12 total, 3 up, 11 in 2024-08-06T17:51:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: pgmap v383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:51:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:54 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:51:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:51:54.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T17:51:54.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:51:54.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:54.740 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:54.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:51:54.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:51:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:51:54.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:54.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:54.975 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:54.975 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:55.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:55.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:51:55.266 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:51:55.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:51:55.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:51:55.283 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:51:55.283 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:51:55.283 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:55 smithi086 ceph-mon[29229]: osdmap e131: 12 total, 3 up, 11 in 2024-08-06T17:51:55.284 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:51:55.284 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:51:55.284 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:51:55.300 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:51:55.301 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:51:55.301 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:51:55.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:51:55.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:51:55.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:51:55.303 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:51:55.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:51:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:55 smithi117 ceph-mon[32835]: osdmap e131: 12 total, 3 up, 11 in 2024-08-06T17:51:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:55 smithi107 ceph-mon[32381]: osdmap e131: 12 total, 3 up, 11 in 2024-08-06T17:51:55.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:55.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:55.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:55.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:55.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:55.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:51:55.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:56.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:56.144 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:56.144 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:56.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:56.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:51:56.202 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:51:56.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:56.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2292632548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:56.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: pgmap v385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:56.315 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:56 smithi117 ceph-mon[32835]: Health check update: Degraded data redundancy: 15 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:56.397 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:51:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T17:51:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T17:51:56.397 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2292632548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: pgmap v385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:56.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:56 smithi107 ceph-mon[32381]: Health check update: Degraded data redundancy: 15 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:56.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:51:56.447 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:51:56.447 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2292632548' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: pgmap v385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:56.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:56 smithi086 ceph-mon[29229]: Health check update: Degraded data redundancy: 15 pgs undersized (PG_DEGRADED) 2024-08-06T17:51:56.481 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:56.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:56.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:56.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:56.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:57.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:57.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T17:51:57.204 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.2 to 1 (10000) 2024-08-06T17:51:57.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:57.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:57.365 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:51:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-06T17:51:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:57 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:51:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:57 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:51:57.376 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-06T17:51:57.376 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:57 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:51:57.376 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:57 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:51:57.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:57.554 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd={"prefix": "osd reweight", "id": 2, "weight": 1.0} : dispatch 2024-08-06T17:51:57.554 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:57 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:51:57.554 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:57 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:51:57.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:57.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:57.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:57.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:57.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:57.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:57.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:58.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:58.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:58.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:51:58.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:51:58.305 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-06T17:51:58.306 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:58 smithi107 ceph-mon[32381]: osdmap e132: 12 total, 3 up, 11 in 2024-08-06T17:51:58.306 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:58 smithi107 ceph-mon[32381]: pgmap v387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:58.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:58.417 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-06T17:51:58.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:58 smithi086 ceph-mon[29229]: osdmap e132: 12 total, 3 up, 11 in 2024-08-06T17:51:58.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:58 smithi086 ceph-mon[29229]: pgmap v387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:58.441 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2681052512' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-08-06T17:51:58.441 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:58 smithi117 ceph-mon[32835]: osdmap e132: 12 total, 3 up, 11 in 2024-08-06T17:51:58.441 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:58 smithi117 ceph-mon[32835]: pgmap v387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:51:58.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 4 1 2024-08-06T17:51:58.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:51:58.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:51:58.713 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:51:58.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:58.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:59.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:59.101 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:51:59.101 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T17:51:59.101 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T17:51:59.102 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T17:51:59.102 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:51:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:51:59.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:51:59.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:51:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:51:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:59.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:51:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:59.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:51:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:51:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:51:59.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:51:59.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:51:59.856 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:51:59.856 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:51:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:51:59.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T17:51:59.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:00.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:00.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:00.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:51:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:00.079 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:00.131 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:52:00.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:00.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:00.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:00 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:00.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:00.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:00 smithi086 ceph-mon[29229]: osdmap e133: 12 total, 3 up, 11 in 2024-08-06T17:52:00.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:00 smithi086 ceph-mon[29229]: pgmap v389: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:00.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:00.542 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:00 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:00.542 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:00.542 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:00 smithi107 ceph-mon[32381]: osdmap e133: 12 total, 3 up, 11 in 2024-08-06T17:52:00.542 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:00 smithi107 ceph-mon[32381]: pgmap v389: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:00.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:00.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:00.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:00 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1194285282' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:00 smithi117 ceph-mon[32835]: osdmap e133: 12 total, 3 up, 11 in 2024-08-06T17:52:00.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:00 smithi117 ceph-mon[32835]: pgmap v389: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:00.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:00.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:00.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:01.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:01.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:01.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:01.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:01.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:01.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:01.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:01.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:01.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:01.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:01.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:01.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head, version: 16'91, data_included: [0~4096,65536~197152], data_size: 201248, omap_header_size: 0, omap_entries_size: 0, attrset_size: 9, recovery_info: ObjectRecoveryInfo(1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head@16'91, size: 262688, copy_subset: [(0, 262688)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 262688, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:47:03.659944+0000", 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.09876099100001, 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00087703499999999997, 2024-08-06T17:52:01.758 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.659944+0000", 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.659944+0000", 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.759 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.659945+0000", 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.9500000000000003e-07 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660204+0000", 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00025905600000000002 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T17:52:01.760 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660218+0000", 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4459e-05 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660230+0000", 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1635e-05 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660309+0000", 2024-08-06T17:52:01.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9185999999999997e-05 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660324+0000", 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.4884e-05 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.660821+0000", 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00049731999999999999 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:52:01.762 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDPGPush(1.0 47/46 [PushOp(1:30306672:devicehealth::main.db.0000000000000000:head, version: 16'92, data_included: [0~327680], data_size: 327680, omap_header_size: 0, omap_entries_size: 0, attrset_size: 10, recovery_info: ObjectRecoveryInfo(1:30306672:devicehealth::main.db.0000000000000000:head@16'92, size: 327680, copy_subset: [(0, 327680)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 327680, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:47:03.662231+0000", 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 298.09647371699998, 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.001084694, 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662231+0000", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662231+0000", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662232+0000", 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8700000000000005e-07 2024-08-06T17:52:01.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662726+0000", 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000493423 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662742+0000", 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6558999999999998e-05 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662752+0000", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0003999999999999e-05 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662831+0000", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9024999999999996e-05 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.662843+0000", 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.184e-05 2024-08-06T17:52:01.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:47:03.663316+0000", 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00047315600000000002 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:131,from:0,reservation_nonce:0)", 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:51:56.969683+0000", 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.789022428, 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023437499999999999, 2024-08-06T17:52:01.767 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969683+0000", 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969683+0000", 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969684+0000", 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.274e-06 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969688+0000", 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.2250000000000002e-06 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969700+0000", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1515000000000001e-05 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969708+0000", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.7500000000000003e-06 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969751+0000", 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3293e-05 2024-08-06T17:52:01.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969769+0000", 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7765e-05 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.969917+0000", 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000148553 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:51:56.970450+0000", 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 4.7882551360000001, 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6309000000000004e-05, 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970450+0000", 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970450+0000", 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970451+0000", 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4799999999999998e-07 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970454+0000", 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0589999999999998e-06 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970461+0000", 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6590000000000007e-06 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970465+0000", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.0820000000000001e-06 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970501+0000", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5525999999999998e-05 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970508+0000", 2024-08-06T17:52:01.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8709999999999999e-06 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:51:56.970526+0000", 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8564e-05 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T17:52:01.776 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:52:01.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:02.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:02.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:02.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:02.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:02.436 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:02.437 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:02.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:52:02.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:02.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T17:52:02.644 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:02.645 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:02.645 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:02.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:02.662 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:02.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:02.663 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:02.663 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:02.679 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:02.680 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:02.683 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.4 to 1 (10000) 2024-08-06T17:52:02.684 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:02.684 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:02.689 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:02.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:02 smithi086 ceph-mon[29229]: pgmap v390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:02.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1873098102' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:02.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:02 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:02.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3893687858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:02.690 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:02.690 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:02.690 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:02.690 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:02.691 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:02.691 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:02.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:02.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:02.853 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:52:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:02 smithi107 ceph-mon[32381]: pgmap v390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1873098102' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:02 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3893687858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:02.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:02.990 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:02.990 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:03.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:02 smithi117 ceph-mon[32835]: pgmap v390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1873098102' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:02 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd reweight", "id": 4, "weight": 1.0} : dispatch 2024-08-06T17:52:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3893687858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:03.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:03.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:03.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:03.547 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:03.548 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:03.581 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:03.581 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:03.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:03.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: osdmap e134: 12 total, 3 up, 11 in 2024-08-06T17:52:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:03 smithi086 ceph-mon[29229]: pgmap v392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-06T17:52:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: osdmap e134: 12 total, 3 up, 11 in 2024-08-06T17:52:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:03 smithi117 ceph-mon[32835]: pgmap v392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:03.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: osdmap e134: 12 total, 3 up, 11 in 2024-08-06T17:52:03.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:03 smithi107 ceph-mon[32381]: pgmap v392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:04.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:04.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 5 1 2024-08-06T17:52:04.152 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:04.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:04.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:04.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:04.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:04.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:04.497 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:04.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:04.587 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:04.588 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:04.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:04.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:04.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:05.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:05.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T17:52:05.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:05.294 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:05.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:05.440 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:52:05.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:05.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:52:05.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:05.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:05.899 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:05.899 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:05.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:06.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:06.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:06.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:06.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:06.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:06.604 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:06.605 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:06.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:06.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:06 smithi086 ceph-mon[29229]: pgmap v393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:06.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:06 smithi086 ceph-mon[29229]: osdmap e135: 12 total, 3 up, 11 in 2024-08-06T17:52:06.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:06.748 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:06.749 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:06.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:06 smithi117 ceph-mon[32835]: pgmap v393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:06 smithi117 ceph-mon[32835]: osdmap e135: 12 total, 3 up, 11 in 2024-08-06T17:52:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:06.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:06 smithi107 ceph-mon[32381]: pgmap v393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:06.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:06 smithi107 ceph-mon[32381]: osdmap e135: 12 total, 3 up, 11 in 2024-08-06T17:52:06.880 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:06.883 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:06.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:07.007 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:07.007 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:07.116 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:07.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:07.175 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:52:07.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:07.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:07.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:07.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:07.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:07.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:07.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:07.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:07.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T17:52:07.830 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:07.830 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:07 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:07 smithi086 ceph-mon[29229]: osdmap e136: 12 total, 3 up, 11 in 2024-08-06T17:52:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:07 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:52:07.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:07 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:52:07.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:07.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:07.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:07.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:07 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:07.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:07.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:07 smithi107 ceph-mon[32381]: osdmap e136: 12 total, 3 up, 11 in 2024-08-06T17:52:07.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:07 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:52:07.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:07 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:52:08.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:08.066 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:52:08.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:07 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:08.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3527386965' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:08.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:07 smithi117 ceph-mon[32835]: osdmap e136: 12 total, 3 up, 11 in 2024-08-06T17:52:08.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:07 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:52:08.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:07 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:52:08.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:08.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:08.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:08.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:08.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:08.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:08.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:08.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:08 smithi107 ceph-mon[32381]: pgmap v396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:08.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:08 smithi107 ceph-mon[32381]: osdmap e137: 12 total, 3 up, 11 in 2024-08-06T17:52:08.732 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-06T17:52:08.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:08.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:08 smithi086 ceph-mon[29229]: pgmap v396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:08.871 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:08 smithi086 ceph-mon[29229]: osdmap e137: 12 total, 3 up, 11 in 2024-08-06T17:52:08.871 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-06T17:52:08.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:08.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:08 smithi117 ceph-mon[32835]: pgmap v396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:08.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:08 smithi117 ceph-mon[32835]: osdmap e137: 12 total, 3 up, 11 in 2024-08-06T17:52:08.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd={"prefix": "osd reweight", "id": 5, "weight": 1.0} : dispatch 2024-08-06T17:52:09.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:09.186 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.5 to 1 (10000) 2024-08-06T17:52:09.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:09.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:09.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:09.590 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:09.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:09.685 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:09.685 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:09.714 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:09.715 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:09.731 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:09.732 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:09.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:09.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:09.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:09.749 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:09.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:09.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:09.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:10.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:10.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 6 1 2024-08-06T17:52:10.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:10.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:10.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T17:52:10.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: osdmap e138: 12 total, 3 up, 11 in 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: pgmap v399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3879099193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: osdmap e138: 12 total, 3 up, 11 in 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: pgmap v399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3879099193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.524 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:10.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3500576065' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: osdmap e138: 12 total, 3 up, 11 in 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: pgmap v399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3879099193' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:10.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:10.839 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:10.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:10.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:10.888 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:11.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:11.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:52:11.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:11.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:11.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:11.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:11.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:11.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:11.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:11 smithi107 ceph-mon[32381]: osdmap e139: 12 total, 3 up, 11 in 2024-08-06T17:52:11.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:11.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:11 smithi117 ceph-mon[32835]: osdmap e139: 12 total, 3 up, 11 in 2024-08-06T17:52:11.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:11.596 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:11.596 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:11 smithi086 ceph-mon[29229]: osdmap e139: 12 total, 3 up, 11 in 2024-08-06T17:52:11.605 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:11.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:11.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:11.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:12.025 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:12.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:12.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:12.262 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:12 smithi086 ceph-mon[29229]: pgmap v401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:12.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:12 smithi107 ceph-mon[32381]: pgmap v401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:12.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:12.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:12 smithi117 ceph-mon[32835]: pgmap v401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:12.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:12.674 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:12.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T17:52:12.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:12.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:52:12.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:12.924 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:13.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:13.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:13.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:13.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:13.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:13.423 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.6 to 1 (10000) 2024-08-06T17:52:13.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:13.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-06T17:52:13.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:13.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-06T17:52:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:13.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:13.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:13.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd={"prefix": "osd reweight", "id": 6, "weight": 1.0} : dispatch 2024-08-06T17:52:13.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:14.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:14.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:14.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:14.391 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:14.428 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:14.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-06T17:52:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:14 smithi086 ceph-mon[29229]: osdmap e140: 12 total, 3 up, 11 in 2024-08-06T17:52:14.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:14 smithi086 ceph-mon[29229]: pgmap v403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-06T17:52:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:14 smithi117 ceph-mon[32835]: osdmap e140: 12 total, 3 up, 11 in 2024-08-06T17:52:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:14 smithi117 ceph-mon[32835]: pgmap v403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:14.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:14.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1329154701' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2024-08-06T17:52:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:14 smithi107 ceph-mon[32381]: osdmap e140: 12 total, 3 up, 11 in 2024-08-06T17:52:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:14 smithi107 ceph-mon[32381]: pgmap v403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:14.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 7 1 2024-08-06T17:52:14.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:14.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:15.010 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:15.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T17:52:15.064 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:15.064 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:15.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:15.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:15.301 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:15.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:15.439 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:52:15.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:15.509 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:15.509 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:15.509 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:15.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:15.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:15.762 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:15.883 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:16.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:16.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:16.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:16.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:16.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:16.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:16.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:16.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:16 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:16.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:16.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:16 smithi117 ceph-mon[32835]: osdmap e141: 12 total, 3 up, 11 in 2024-08-06T17:52:16.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:16 smithi117 ceph-mon[32835]: pgmap v405: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:16.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:16.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:16 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:16.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:16.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:16 smithi086 ceph-mon[29229]: osdmap e141: 12 total, 3 up, 11 in 2024-08-06T17:52:16.704 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:16 smithi086 ceph-mon[29229]: pgmap v405: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:16 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2203232947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:16 smithi107 ceph-mon[32381]: osdmap e141: 12 total, 3 up, 11 in 2024-08-06T17:52:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:16 smithi107 ceph-mon[32381]: pgmap v405: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:16.741 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:16.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:16.932 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:16.946 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:16.947 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:16.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:16.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:16.969 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:16.969 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:16.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:16.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:16.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:16.988 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:16.988 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:16.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:16.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:16.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:16.990 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:16.990 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:17.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:17.108 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:17.108 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:17.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:17.340 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:17.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T17:52:17.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:17.444 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:17.444 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:17.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:17.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3131983958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:17.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:17.593 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3131983958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:17.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3131983958' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:17.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:17.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:18.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:18.085 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:18.085 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:18.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:18.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:18.112 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:18.112 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:18.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:18.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:18.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:18.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:52:18.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:18 smithi107 ceph-mon[32381]: pgmap v406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:18.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:18.610 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:18.610 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:18.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.611 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:18 smithi086 ceph-mon[29229]: pgmap v406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:18.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:18.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:18.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:18.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:18.734 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:18 smithi117 ceph-mon[32835]: pgmap v406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:18.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:18.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:18.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:19.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:19.038 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:19.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:19.185 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:19.185 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:19.186 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:19.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:19.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:19.453 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.7 to 1 (10000) 2024-08-06T17:52:19.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:19.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:19.681 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-06T17:52:19.700 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-06T17:52:19.724 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:19.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T17:52:19.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:19.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd={"prefix": "osd reweight", "id": 7, "weight": 1.0} : dispatch 2024-08-06T17:52:19.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:19.968 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:20.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:20.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:52:20.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:20.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:20.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 8 1 2024-08-06T17:52:20.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:20.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:20.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-06T17:52:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:20 smithi107 ceph-mon[32381]: osdmap e142: 12 total, 3 up, 11 in 2024-08-06T17:52:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:20 smithi107 ceph-mon[32381]: pgmap v408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:20.730 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-06T17:52:20.731 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:20 smithi117 ceph-mon[32835]: osdmap e142: 12 total, 3 up, 11 in 2024-08-06T17:52:20.731 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:20 smithi117 ceph-mon[32835]: pgmap v408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:20.733 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:20.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1688586045' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-08-06T17:52:20.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:20 smithi086 ceph-mon[29229]: osdmap e142: 12 total, 3 up, 11 in 2024-08-06T17:52:20.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:20 smithi086 ceph-mon[29229]: pgmap v408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:20.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:20.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:21.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:21.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:21.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:21.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:21.521 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:52:21.521 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:52:21.521 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:21 smithi117 ceph-mon[32835]: Health check update: Degraded data redundancy: 16 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:21.521 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:21 smithi117 ceph-mon[32835]: osdmap e143: 12 total, 3 up, 11 in 2024-08-06T17:52:21.552 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:52:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:52:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:21 smithi086 ceph-mon[29229]: Health check update: Degraded data redundancy: 16 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:21 smithi086 ceph-mon[29229]: osdmap e143: 12 total, 3 up, 11 in 2024-08-06T17:52:21.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:21.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:21.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:52:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:52:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:21 smithi107 ceph-mon[32381]: Health check update: Degraded data redundancy: 16 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:21 smithi107 ceph-mon[32381]: osdmap e143: 12 total, 3 up, 11 in 2024-08-06T17:52:21.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:21.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:22.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:22.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:22.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:22.121 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:22.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T17:52:22.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:22.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:22.325 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:22.364 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:22.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:22.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:22.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:22.724 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:22 smithi117 ceph-mon[32835]: pgmap v410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:22.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:22 smithi086 ceph-mon[29229]: pgmap v410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:22.773 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:22.773 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:22.773 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:22.773 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:22.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:22 smithi107 ceph-mon[32381]: pgmap v410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:22.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:23.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:23.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:23.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:23.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:23.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:23.466 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:52:23.486 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:23.487 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:23.487 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:23.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:23.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:23.648 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:23 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:52:23.648 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:23 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:52:23.648 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:23.648 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3901855774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:23.679 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:23.679 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:23.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:23 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:52:23.679 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:23 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:52:23.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:23.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3901855774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:23.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:23.698 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:23.699 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:23.699 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:23.699 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:23.699 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:23.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:23.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:23.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:23.878 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:23.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:23 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:52:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:23 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:52:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3901855774' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:24.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:24.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:24.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:24.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:24.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:24.516 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:24.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T17:52:24.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:52:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: osdmap e144: 12 total, 3 up, 11 in 2024-08-06T17:52:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: pgmap v412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:24.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:24.728 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:52:24.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:24.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: osdmap e144: 12 total, 3 up, 11 in 2024-08-06T17:52:24.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: pgmap v412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:24.729 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.760 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:24.825 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:24.825 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:24.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:52:24.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/460926079' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:24.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: osdmap e144: 12 total, 3 up, 11 in 2024-08-06T17:52:24.826 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: pgmap v412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:24.826 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:24.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:24.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:25.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:25.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:25.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:25.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:25.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:25.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:25.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:25.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:25.579 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:25.580 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:25.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:25.660 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:25.660 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:25.660 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:25.660 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:25 smithi107 ceph-mon[32381]: Health check update: Reduced data availability: 30 pgs inactive, 6 pgs stale (PG_AVAILABILITY) 2024-08-06T17:52:25.660 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-06T17:52:25.709 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.8 to 1 (10000) 2024-08-06T17:52:25.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:25.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:25 smithi117 ceph-mon[32835]: Health check update: Reduced data availability: 30 pgs inactive, 6 pgs stale (PG_AVAILABILITY) 2024-08-06T17:52:25.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-06T17:52:25.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:25 smithi086 ceph-mon[29229]: Health check update: Reduced data availability: 30 pgs inactive, 6 pgs stale (PG_AVAILABILITY) 2024-08-06T17:52:25.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd={"prefix": "osd reweight", "id": 8, "weight": 1.0} : dispatch 2024-08-06T17:52:25.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:26.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:26.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:52:26.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:26.267 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:26.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:26.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 9 1 2024-08-06T17:52:26.373 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:26.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:26.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:26.594 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:26.594 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:26.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-06T17:52:26.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:26 smithi107 ceph-mon[32381]: osdmap e145: 12 total, 3 up, 11 in 2024-08-06T17:52:26.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:26 smithi107 ceph-mon[32381]: pgmap v414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:26.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:26.727 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-06T17:52:26.728 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:26 smithi117 ceph-mon[32835]: osdmap e145: 12 total, 3 up, 11 in 2024-08-06T17:52:26.728 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:26 smithi117 ceph-mon[32835]: pgmap v414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:26.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1030550091' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 8, "weight": 1.0}]': finished 2024-08-06T17:52:26.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:26 smithi086 ceph-mon[29229]: osdmap e145: 12 total, 3 up, 11 in 2024-08-06T17:52:26.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:26 smithi086 ceph-mon[29229]: pgmap v414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:26.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:26.863 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:26.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T17:52:26.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:27.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:27.105 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:27.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:27.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:27.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:27.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:27.453 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:27.453 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:27.457 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:27.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:27.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:52:27.632 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:27 smithi107 ceph-mon[32381]: Health check update: Degraded data redundancy: 18 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:27.717 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:27.718 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:27.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:27 smithi086 ceph-mon[29229]: Health check update: Degraded data redundancy: 18 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:27.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:27 smithi117 ceph-mon[32835]: Health check update: Degraded data redundancy: 18 pgs undersized (PG_DEGRADED) 2024-08-06T17:52:27.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:28.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:28.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:28.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:28.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:28.185 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:28.185 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:28.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:28.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:28.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:28.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:28 smithi086 ceph-mon[29229]: pgmap v415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:28.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:28.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:28.682 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:28.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:28.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:28.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:28 smithi117 ceph-mon[32835]: pgmap v415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:28 smithi107 ceph-mon[32381]: pgmap v415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:28.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:28.995 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:28.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:29.072 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:29.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:29.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:29.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:29.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:29.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:29.405 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:29.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T17:52:29.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:29.501 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:52:29.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:52:29.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:52:29.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:52:29.516 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:52:29.516 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:52:29.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:29.656 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:52:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:52:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:52:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:52:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:52:29.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:29.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:52:29.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:52:29.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:52:29.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:52:29.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:52:29.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:29.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:29.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:30.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:30.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:30.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:30.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:30.510 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.9 to 1 (10000) 2024-08-06T17:52:30.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:30.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:30.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:30 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:30.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:30.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:30 smithi107 ceph-mon[32381]: osdmap e146: 12 total, 3 up, 11 in 2024-08-06T17:52:30.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:30 smithi107 ceph-mon[32381]: pgmap v417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:30.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-06T17:52:30.725 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:30 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:30.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:30.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:30 smithi117 ceph-mon[32835]: osdmap e146: 12 total, 3 up, 11 in 2024-08-06T17:52:30.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:30 smithi117 ceph-mon[32835]: pgmap v417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:30.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-06T17:52:30.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:30.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:30 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:30.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3508198924' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:52:30.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:30 smithi086 ceph-mon[29229]: osdmap e146: 12 total, 3 up, 11 in 2024-08-06T17:52:30.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:30 smithi086 ceph-mon[29229]: pgmap v417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:30.818 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd={"prefix": "osd reweight", "id": 9, "weight": 1.0} : dispatch 2024-08-06T17:52:30.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:31.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:31.112 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:31.112 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:31.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:31.220 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:31.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:31.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:31.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:31.620 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-06T17:52:31.620 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:31 smithi117 ceph-mon[32835]: osdmap e147: 12 total, 3 up, 11 in 2024-08-06T17:52:31.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:31.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T17:52:31.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-06T17:52:31.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:31 smithi086 ceph-mon[29229]: osdmap e147: 12 total, 3 up, 11 in 2024-08-06T17:52:31.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 10 1 2024-08-06T17:52:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1622927874' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 9, "weight": 1.0}]': finished 2024-08-06T17:52:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:31 smithi107 ceph-mon[32381]: osdmap e147: 12 total, 3 up, 11 in 2024-08-06T17:52:32.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:32.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:32.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:32.135 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:32.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:32.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:32.419 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:32.419 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:32.419 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:32.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:32.430 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:32.437 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:32.437 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:32.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:32.438 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:32.438 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:32.456 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:32.458 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:32.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:32.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:32.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:32.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:32 smithi086 ceph-mon[29229]: pgmap v419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:32.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1521264843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:32.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:32.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:32.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:32 smithi107 ceph-mon[32381]: pgmap v419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1521264843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:32.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:32.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:52:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:32 smithi117 ceph-mon[32835]: pgmap v419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1521264843' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:32.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:33.032 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:33.033 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:33.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:33.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:33.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:33.445 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:33.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:33.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:33.627 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:33.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:33.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:33.788 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:33.788 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:33.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:33.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:33.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:33.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:33.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:34.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:34.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:34.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:34.327 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:34.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T17:52:34.369 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:34.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:34.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:34.603 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:34 smithi086 ceph-mon[29229]: pgmap v420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:34.618 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:34.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:34 smithi117 ceph-mon[32835]: pgmap v420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:34.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:34.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:34.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:34.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:34 smithi107 ceph-mon[32381]: pgmap v420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:35.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:35.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:35.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:35.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:52:35.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:35.528 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.10 to 1 (10000) 2024-08-06T17:52:35.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:35.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-06T17:52:35.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-06T17:52:35.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:35.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:35.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:35.914 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd={"prefix": "osd reweight", "id": 10, "weight": 1.0} : dispatch 2024-08-06T17:52:36.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:36.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:36.185 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:36.185 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:36.208 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:36.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:36.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:36.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd reweight 11 1 2024-08-06T17:52:36.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-06T17:52:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:36 smithi117 ceph-mon[32835]: osdmap e148: 12 total, 3 up, 11 in 2024-08-06T17:52:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:36 smithi117 ceph-mon[32835]: pgmap v422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:36 smithi117 ceph-mon[32835]: osdmap e149: 12 total, 3 up, 11 in 2024-08-06T17:52:36.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:36.792 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:36.792 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:36.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-06T17:52:36.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:36 smithi086 ceph-mon[29229]: osdmap e148: 12 total, 3 up, 11 in 2024-08-06T17:52:36.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:36 smithi086 ceph-mon[29229]: pgmap v422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:36.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:36 smithi086 ceph-mon[29229]: osdmap e149: 12 total, 3 up, 11 in 2024-08-06T17:52:36.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:36.916 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:36.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T17:52:36.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3038667802' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 10, "weight": 1.0}]': finished 2024-08-06T17:52:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:36 smithi107 ceph-mon[32381]: osdmap e148: 12 total, 3 up, 11 in 2024-08-06T17:52:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:36 smithi107 ceph-mon[32381]: pgmap v422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:36 smithi107 ceph-mon[32381]: osdmap e149: 12 total, 3 up, 11 in 2024-08-06T17:52:37.071 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:37.072 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:37.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:37.114 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:37.114 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:37.166 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:52:37.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:37.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:37.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:37.540 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:52:37.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:37.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:37.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:37.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:38.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:38.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:38.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:38.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:38.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:38.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:38.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:38.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:38 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:38.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:38.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:38 smithi086 ceph-mon[29229]: osdmap e150: 12 total, 3 up, 11 in 2024-08-06T17:52:38.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:38 smithi086 ceph-mon[29229]: pgmap v425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:38.648 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:38.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:38.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:38.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:38 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:38.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:38.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:38 smithi117 ceph-mon[32835]: osdmap e150: 12 total, 3 up, 11 in 2024-08-06T17:52:38.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:38 smithi117 ceph-mon[32835]: pgmap v425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:38.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:38.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:38 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/861988715' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:38 smithi107 ceph-mon[32381]: osdmap e150: 12 total, 3 up, 11 in 2024-08-06T17:52:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:38 smithi107 ceph-mon[32381]: pgmap v425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:39.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:39.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:39.276 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:39.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T17:52:39.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:39.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:39.577 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:39.577 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:39.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:39.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:39 smithi086 ceph-mon[29229]: pgmap v426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:39.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:39.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:39.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:39.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:39 smithi117 ceph-mon[32835]: pgmap v426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:39 smithi107 ceph-mon[32381]: pgmap v426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:40.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:40.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:40.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:40.393 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:40.393 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:40.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:40.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:40.410 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:40.410 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:40.410 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:40.410 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:40.411 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:40.427 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:40.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:40.428 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:40.428 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:40.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:40.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:40.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:40.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:40.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:40.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:40.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:40.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1987635670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:40.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:40.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:40.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:41.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:52:41.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:41.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:41.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1987635670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:41.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1987635670' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:41.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:41.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:41.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:41.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:41.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:41.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:41.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:41.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:41.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:41.527 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:41.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:41.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:41.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:41.756 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:41.756 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:41.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:41 smithi086 ceph-mon[29229]: osdmap e151: 12 total, 3 up, 11 in 2024-08-06T17:52:41.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-06T17:52:41.756 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:41 smithi086 ceph-mon[29229]: pgmap v428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:41.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:41 smithi107 ceph-mon[32381]: osdmap e151: 12 total, 3 up, 11 in 2024-08-06T17:52:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-06T17:52:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:41 smithi107 ceph-mon[32381]: pgmap v428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:42.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:41 smithi117 ceph-mon[32835]: osdmap e151: 12 total, 3 up, 11 in 2024-08-06T17:52:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd={"prefix": "osd reweight", "id": 11, "weight": 1.0} : dispatch 2024-08-06T17:52:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:41 smithi117 ceph-mon[32835]: pgmap v428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:42.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:42.184 INFO:teuthology.orchestra.run.smithi086.stderr:reweighted osd.11 to 1 (10000) 2024-08-06T17:52:42.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:42.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:42.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:42.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:42.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:42.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:42.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:42.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:42.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:42.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:43.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:43.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:43.305 INFO:tasks.thrashosds.thrasher:test_map_discontinuity 2024-08-06T17:52:43.306 INFO:tasks.thrashosds.thrasher:Waiting for recovery 2024-08-06T17:52:43.306 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-06T17:52:43.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:52:43.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-06T17:52:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:43 smithi117 ceph-mon[32835]: osdmap e152: 12 total, 3 up, 11 in 2024-08-06T17:52:43.405 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-06T17:52:43.405 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:43 smithi086 ceph-mon[29229]: osdmap e152: 12 total, 3 up, 11 in 2024-08-06T17:52:43.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2749589209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 11, "weight": 1.0}]': finished 2024-08-06T17:52:43.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:43 smithi107 ceph-mon[32381]: osdmap e152: 12 total, 3 up, 11 in 2024-08-06T17:52:43.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:43.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:52:43.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:43.657 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:43.657 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:43.738 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:43.747 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:43.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:43.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:44.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:44.049 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:44.049 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:44.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:44.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:44.319 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:44 smithi107 ceph-mon[32381]: pgmap v430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:44.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:44 smithi086 ceph-mon[29229]: pgmap v430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:44.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:44.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:44.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:44.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:44.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:44 smithi117 ceph-mon[32835]: pgmap v430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:44.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:44.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:44.941 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:44.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:44.945 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:44.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:44.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:44.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:44.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:44.979 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:44.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T17:52:45.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:45.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:45.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:45.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:45.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:45.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:45.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:45.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:45.807 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:45 smithi107 ceph-mon[32381]: pgmap v431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:45 smithi086 ceph-mon[29229]: pgmap v431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:45.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:46.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:46.076 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:46.076 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:45 smithi117 ceph-mon[32835]: pgmap v431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:46.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:46.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:46.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:46.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:46.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:46.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:46.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:47.064 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:47.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:47.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:47.294 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:47.294 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:47.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:47 smithi086 ceph-mon[29229]: osdmap e153: 12 total, 3 up, 11 in 2024-08-06T17:52:47.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:47 smithi107 ceph-mon[32381]: osdmap e153: 12 total, 3 up, 11 in 2024-08-06T17:52:47.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:47 smithi117 ceph-mon[32835]: osdmap e153: 12 total, 3 up, 11 in 2024-08-06T17:52:47.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:47.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:47.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:48.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:48.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:48.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:48.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:48.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:48.380 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:48 smithi107 ceph-mon[32381]: pgmap v433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:48.381 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:48.381 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:48.492 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:48 smithi086 ceph-mon[29229]: pgmap v433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:48.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:48 smithi117 ceph-mon[32835]: pgmap v433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:48.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:48.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:48.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:48.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:48.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:49.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:49.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:49.160 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:49.161 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:49.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:49.178 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:49.179 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:49.179 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:49.179 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:49.179 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:49.194 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:52:49.197 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:49.197 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:49.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:49.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/876557938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:49.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:49.453 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:49.453 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/876557938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:49.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:52:49.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/876557938' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:49.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:49.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:49.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:49.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:50.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:50.101 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:50.102 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:50.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:50.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:50.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:50.255 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: osdmap e154: 12 total, 3 up, 11 in 2024-08-06T17:52:50.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.256 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:50 smithi117 ceph-mon[32835]: pgmap v435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:50.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:50.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: osdmap e154: 12 total, 3 up, 11 in 2024-08-06T17:52:50.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:50 smithi086 ceph-mon[29229]: pgmap v435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:50.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1743904489' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: osdmap e154: 12 total, 3 up, 11 in 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:50.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:50 smithi107 ceph-mon[32381]: pgmap v435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:50.489 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:52:50.489 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":154,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:52:49.190588+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:07.203826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:51:53.847656+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:52:50.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:50.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:50.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:50.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:50.805 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:50.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:51.025 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:51.026 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:51.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:51.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:51.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:51.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1122135066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:52:51.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:51 smithi086 ceph-mon[29229]: osdmap e155: 12 total, 3 up, 11 in 2024-08-06T17:52:51.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:51.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1122135066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:52:51.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:51 smithi117 ceph-mon[32835]: osdmap e155: 12 total, 3 up, 11 in 2024-08-06T17:52:51.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1122135066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:52:51.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:51 smithi107 ceph-mon[32381]: osdmap e155: 12 total, 3 up, 11 in 2024-08-06T17:52:51.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:51.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:51.756 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:51.756 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:51.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:51.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:52.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:52.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:52.251 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:52 smithi117 ceph-mon[32835]: pgmap v437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:52.265 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:52:52.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:52 smithi086 ceph-mon[29229]: pgmap v437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:52.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:52.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:52 smithi107 ceph-mon[32381]: pgmap v437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 304 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:52.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:52.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:52.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:52.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:52:52.836 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:52:52.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T17:52:52.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:53.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:53.064 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:53.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:53.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:53.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:52:53.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:53.438 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:53.438 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:53.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:53.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:53.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:53.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:53.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:53 smithi086 ceph-mon[29229]: pgmap v438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:54.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:53 smithi117 ceph-mon[32835]: pgmap v438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:53 smithi107 ceph-mon[32381]: pgmap v438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:54.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:54.263 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:54.263 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:54.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:54.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:54.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:54.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:54.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:54.697 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:52:54.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:52:54.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:52:54.702 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:54.702 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:54.716 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:52:54.716 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:52:54.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:52:54.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:52:54.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:52:54.735 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:52:54.736 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:52:54.736 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:52:54.736 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:52:54.737 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:52:54.737 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:52:54.737 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:52:54.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:54.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2325342622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:54.848 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:52:54.848 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:52:54.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:55.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:55.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:52:55.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:55.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:55.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:55.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2325342622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:55.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:55.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2325342622' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:52:55.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:52:55.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:55.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:55.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:55.750 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:55.750 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:55.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:55.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:55.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:55.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:55.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:55.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:55 smithi107 ceph-mon[32381]: pgmap v439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:56.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:55 smithi086 ceph-mon[29229]: pgmap v439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:56.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:52:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:55 smithi117 ceph-mon[32835]: pgmap v439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:56.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:56.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:56.384 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:52:56.385 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:52:56.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:56.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:56.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:56.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:56.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:52:56.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:57.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:57.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:57.277 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:57.278 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:57.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:57.331 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:52:57.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:57.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:57.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:57.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:57 smithi086 ceph-mon[29229]: pgmap v440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:57.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:58.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:57 smithi107 ceph-mon[32381]: pgmap v440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:58.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:57 smithi117 ceph-mon[32835]: pgmap v440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:58.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:52:58.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:52:58.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:52:58.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:52:58.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:52:58.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:58.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:58.754 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:52:58.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/964613110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:58 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:58.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:52:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/964613110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:58 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:59.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:52:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/964613110' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:58 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:52:59.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:52:59.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:52:59.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:52:59.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:52:59.605 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:52:59.606 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:52:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:52:59.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:52:59.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:52:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:59 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:52:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:59 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:52:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:59 smithi107 ceph-mon[32381]: osdmap e156: 12 total, 3 up, 11 in 2024-08-06T17:52:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:52:59 smithi107 ceph-mon[32381]: pgmap v442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:52:59.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:00.001 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:00.001 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:52:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:00.002 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:00.002 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:52:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:00.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:59 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:53:00.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:59 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:00.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:59 smithi086 ceph-mon[29229]: osdmap e156: 12 total, 3 up, 11 in 2024-08-06T17:53:00.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:52:59 smithi086 ceph-mon[29229]: pgmap v442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:00.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:59 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:53:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:59 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:59 smithi117 ceph-mon[32835]: osdmap e156: 12 total, 3 up, 11 in 2024-08-06T17:53:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:52:59 smithi117 ceph-mon[32835]: pgmap v442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:00.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:00.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:00.439 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:53:00.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:00.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:00.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:00.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:01.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:01.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:01.185 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:01.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:01.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:01 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:53:01.308 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:01 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:53:01.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":157,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:01.186615+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:07.203826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:51:53.847656+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:01.333 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:01 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:53:01.333 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:01 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:53:01.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:01 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:53:01.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:01 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:53:01.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:01.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:53:01.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:53:01.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T17:53:01.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:01.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:01.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:02.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:02.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:02.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:02 smithi086 ceph-mon[29229]: osdmap e157: 12 total, 3 up, 11 in 2024-08-06T17:53:02.259 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2851351500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:02.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:02 smithi086 ceph-mon[29229]: pgmap v444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:02.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:02.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:02.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:02.403 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:02 smithi107 ceph-mon[32381]: osdmap e157: 12 total, 3 up, 11 in 2024-08-06T17:53:02.403 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2851351500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:02.403 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:02 smithi107 ceph-mon[32381]: pgmap v444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:02.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:02.603 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:02.604 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:02 smithi117 ceph-mon[32835]: osdmap e157: 12 total, 3 up, 11 in 2024-08-06T17:53:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2851351500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:02 smithi117 ceph-mon[32835]: pgmap v444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:02.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:02.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:02.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:02.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:02.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:03.011 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:03.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:03.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:03.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:03.475 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:03.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:03.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:03.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:03.798 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:03 smithi107 ceph-mon[32381]: pgmap v445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:03.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:04.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:04.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:03 smithi086 ceph-mon[29229]: pgmap v445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:04.045 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:04.045 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:04.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:03 smithi117 ceph-mon[32835]: pgmap v445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:04.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:04.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:04.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:04.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:04.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:04.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:04.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:04.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:04.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:04.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:05.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:05.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:05.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:05.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:05.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:05.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:05.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:05.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:05.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:05.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:05 smithi086 ceph-mon[29229]: pgmap v446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:05.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:05.990 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:05.991 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:05.991 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:06.008 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:06.008 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:06.009 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:06.009 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:06.009 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:06.027 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:06.028 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:06.028 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:06.028 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:06.029 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:06.029 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:06.029 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:06.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:05 smithi117 ceph-mon[32835]: pgmap v446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:06.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:05 smithi107 ceph-mon[32381]: pgmap v446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:06.244 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:53:06.244 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:53:06.244 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:06.244 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:06.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:06.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:06.493 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:06.493 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:06.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:06.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:06.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:06.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T17:53:06.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3870675308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:06.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:06.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:07.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3870675308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:07.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.033 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3870675308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:07.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:07.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:07.469 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:07.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:07.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:07.563 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:07.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:07.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:07.839 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:07 smithi086 ceph-mon[29229]: pgmap v447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:07.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:07 smithi107 ceph-mon[32381]: pgmap v447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:08.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:07 smithi117 ceph-mon[32835]: pgmap v447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:08.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:08.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:08.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:08.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:08.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:08.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:08.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:08.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:08.887 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:08.887 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:08.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:09.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:09.223 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:09.223 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:09.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:09.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:09.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:53:09.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:09.667 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:53:09.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:09.674 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:09.674 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:09.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:09.811 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:09.811 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":158,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:09.666196+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:07.203826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:51:53.847656+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:09.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:09.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:10.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:10.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:10.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:10.358 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:10.358 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:10.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:10.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:10.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:10.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:10.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:10 smithi117 ceph-mon[32835]: pgmap v448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:10.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:10 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:10.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:10.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:10 smithi117 ceph-mon[32835]: osdmap e158: 12 total, 3 up, 11 in 2024-08-06T17:53:10.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3954117348' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:10.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:10.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:10 smithi107 ceph-mon[32381]: pgmap v448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:10 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:10 smithi107 ceph-mon[32381]: osdmap e158: 12 total, 3 up, 11 in 2024-08-06T17:53:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3954117348' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:11.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:11.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:10 smithi086 ceph-mon[29229]: pgmap v448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:11.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:10 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:11.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/914433202' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:11.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:10 smithi086 ceph-mon[29229]: osdmap e158: 12 total, 3 up, 11 in 2024-08-06T17:53:11.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3954117348' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:11.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:11.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:11.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:11.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:11.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:11.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:11.584 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:11.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:11.704 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:11.705 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:11.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:11.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:12.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:12.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:12.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:12.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:12 smithi086 ceph-mon[29229]: osdmap e159: 12 total, 3 up, 11 in 2024-08-06T17:53:12.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:12 smithi086 ceph-mon[29229]: pgmap v451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:12.355 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:12.355 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:12 smithi107 ceph-mon[32381]: osdmap e159: 12 total, 3 up, 11 in 2024-08-06T17:53:12.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:12 smithi107 ceph-mon[32381]: pgmap v451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:12.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:12.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:12.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:12.507 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:12 smithi117 ceph-mon[32835]: osdmap e159: 12 total, 3 up, 11 in 2024-08-06T17:53:12.507 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:12 smithi117 ceph-mon[32835]: pgmap v451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:12.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:12.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:12.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:12.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:12.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:12.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:53:13.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:13.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:13.202 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:13.202 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:13.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:13.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:13.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:13.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:13.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:13.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:13.846 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:13.847 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:13.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:13.857 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:13.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:13.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:13.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:13.874 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:13.876 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:13.876 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:13.876 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:13.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:13.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:13.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:13.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:13.964 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:53:13.964 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:53:14.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:14.115 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:13 smithi107 ceph-mon[32381]: pgmap v452: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:13 smithi086 ceph-mon[29229]: pgmap v452: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:13 smithi117 ceph-mon[32835]: pgmap v452: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:14.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:14.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:14.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:14.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:14.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:14.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T17:53:14.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:14.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/962308964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:14.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/962308964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:15.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/962308964' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:15.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:15.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:15.198 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:15.198 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:15.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:15.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:15.390 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:15.390 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:15.390 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:15.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:15.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:15.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:15.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:15.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:15.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:15.833 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:15.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:15 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:53:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:15 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:53:15.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:15 smithi086 ceph-mon[29229]: pgmap v453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:15.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:15.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:16.005 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:15 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:53:16.005 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:15 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:53:16.005 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:15 smithi107 ceph-mon[32381]: pgmap v453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:16.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:15 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:53:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:15 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:53:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:15 smithi117 ceph-mon[32835]: pgmap v453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:16.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:16.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:16.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:16.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:16.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:16.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:16.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:16.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:16 smithi086 ceph-mon[29229]: osdmap e160: 12 total, 3 up, 11 in 2024-08-06T17:53:16.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:16 smithi086 ceph-mon[29229]: osdmap e161: 12 total, 3 up, 11 in 2024-08-06T17:53:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:16 smithi117 ceph-mon[32835]: osdmap e160: 12 total, 3 up, 11 in 2024-08-06T17:53:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:16 smithi117 ceph-mon[32835]: osdmap e161: 12 total, 3 up, 11 in 2024-08-06T17:53:16.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:16.946 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:16.946 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:16.946 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:16 smithi107 ceph-mon[32381]: osdmap e160: 12 total, 3 up, 11 in 2024-08-06T17:53:16.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:16 smithi107 ceph-mon[32381]: osdmap e161: 12 total, 3 up, 11 in 2024-08-06T17:53:17.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:17.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:17.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:17.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:17.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:53:17.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:17.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:17.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:17.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:17.740 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:17.741 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:17.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:17.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:18.013 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:17 smithi107 ceph-mon[32381]: pgmap v456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:18.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:18.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:17 smithi086 ceph-mon[29229]: pgmap v456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:18.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:18.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:18.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:18.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:17 smithi117 ceph-mon[32835]: pgmap v456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:18.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:18.278 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:18.278 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:18.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:18.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:18.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:18.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:18.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:18.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:18.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:19.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:19.086 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3595637891' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:19.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3595637891' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:19.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3595637891' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:19.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:19.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:19.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:19.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:19.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:39 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'21, mlcod=16'19)", 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:08.912885+0000", 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.01697663499999, 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068260399999999996, 2024-08-06T17:53:19.930 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912885+0000", 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912885+0000", 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912885+0000", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4300000000000003e-07 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912891+0000", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1399999999999997e-06 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912900+0000", 2024-08-06T17:53:19.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0990000000000005e-06 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912911+0000", 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0482999999999999e-05 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912959+0000", 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8106000000000003e-05 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.912977+0000", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8403e-05 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.913524+0000", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00054649899999999999 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.913567+0000", 2024-08-06T17:53:19.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3331000000000002e-05 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:46 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'28, mlcod=16'24)", 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:08.916644+0000", 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.01321709299998, 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0013960190000000001, 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916644+0000", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916644+0000", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916645+0000", 2024-08-06T17:53:19.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7700000000000004e-07 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916649+0000", 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7729999999999999e-06 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916660+0000", 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0339e-05 2024-08-06T17:53:19.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916666+0000", 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8989999999999998e-06 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916730+0000", 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.389e-05 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.938 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.916740+0000", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0885e-05 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918006+0000", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0012650599999999999 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918040+0000", 2024-08-06T17:53:19.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4595999999999999e-05 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:48 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'30, mlcod=16'26)", 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:08.918268+0000", 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.01159328, 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001432212, 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.940 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918268+0000", 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918268+0000", 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918269+0000", 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7999999999999995e-07 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918273+0000", 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3549999999999998e-06 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918284+0000", 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0577999999999999e-05 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918290+0000", 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2729999999999999e-06 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918322+0000", 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1866999999999998e-05 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.943 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918340+0000", 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7997000000000001e-05 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.919677+0000", 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001337261 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.919700+0000", 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3201e-05 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.944 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:49 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'31, mlcod=16'26)", 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:08.918590+0000", 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.01127070299998, 2024-08-06T17:53:19.945 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001258604, 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.946 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918590+0000", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918590+0000", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918591+0000", 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7000000000000004e-07 2024-08-06T17:53:19.947 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918596+0000", 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6500000000000004e-06 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918607+0000", 2024-08-06T17:53:19.948 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0791000000000001e-05 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918612+0000", 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6869999999999999e-06 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918650+0000", 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7527000000000002e-05 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.949 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.918665+0000", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5125e-05 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.919824+0000", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.001159167 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.919849+0000", 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4987e-05 2024-08-06T17:53:19.950 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:53 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'35, mlcod=16'33)", 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:08.921557+0000", 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 491.00830417499998, 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00062272, 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.951 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921557+0000", 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921557+0000", 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.952 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921558+0000", 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6000000000000004e-07 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921596+0000", 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8238999999999998e-05 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921605+0000", 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0520000000000007e-06 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.953 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921611+0000", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6779999999999997e-06 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921659+0000", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8158999999999997e-05 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.921676+0000", 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7689999999999998e-05 2024-08-06T17:53:19.954 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.922153+0000", 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00047612499999999999 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:08.922180+0000", 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7217e-05 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.955 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:69 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'47, mlcod=16'43)", 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.059257+0000", 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.870603765, 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00058976299999999998, 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059257+0000", 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059257+0000", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059258+0000", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9200000000000001e-07 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059277+0000", 2024-08-06T17:53:19.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8661999999999999e-05 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059285+0000", 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8589999999999994e-06 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059291+0000", 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7579999999999998e-06 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059340+0000", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8443999999999999e-05 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059360+0000", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9959e-05 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059809+0000", 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000449216 2024-08-06T17:53:19.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.059847+0000", 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8373e-05 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:71 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'49, mlcod=16'45)", 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.060453+0000", 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86940856199999, 2024-08-06T17:53:19.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00071664500000000002, 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060453+0000", 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060453+0000", 2024-08-06T17:53:19.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060453+0000", 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8899999999999999e-07 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060458+0000", 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.639e-06 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060468+0000", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0552e-05 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060474+0000", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.536e-06 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060512+0000", 2024-08-06T17:53:19.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7929000000000002e-05 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060528+0000", 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5710999999999999e-05 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.061136+0000", 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000608914 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.964 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.061169+0000", 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2774999999999997e-05 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:72 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'50, mlcod=16'45)", 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.060742+0000", 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.869119399, 2024-08-06T17:53:19.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00061774300000000002, 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060742+0000", 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060742+0000", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060742+0000", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4499999999999997e-07 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060784+0000", 2024-08-06T17:53:19.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1334000000000001e-05 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060794+0000", 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0641e-05 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060800+0000", 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7780000000000004e-06 2024-08-06T17:53:19.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060842+0000", 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1955000000000001e-05 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.060859+0000", 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6878999999999999e-05 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.061314+0000", 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00045526399999999998 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.061360+0000", 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5346999999999998e-05 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.970 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:75 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'53, mlcod=16'49)", 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.062489+0000", 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86737237900002, 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059256399999999996, 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062489+0000", 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062489+0000", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062489+0000", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2499999999999995e-07 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062509+0000", 2024-08-06T17:53:19.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9780999999999999e-05 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062518+0000", 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2070000000000001e-06 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062524+0000", 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0499999999999997e-06 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062564+0000", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9298000000000002e-05 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.062575+0000", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.149e-05 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.063052+0000", 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000476811 2024-08-06T17:53:19.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.063081+0000", 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9402000000000001e-05 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:79 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'57, mlcod=16'53)", 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.064985+0000", 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86487649499998, 2024-08-06T17:53:19.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000675604, 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.064985+0000", 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.064985+0000", 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.064985+0000", 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7800000000000002e-07 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065022+0000", 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7181000000000002e-05 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065027+0000", 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.386e-06 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065030+0000", 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1209999999999998e-06 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065213+0000", 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018275600000000001 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065229+0000", 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6189999999999999e-05 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065634+0000", 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00040528100000000001 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065660+0000", 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6310999999999999e-05 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:80 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'58, mlcod=16'53)", 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.065246+0000", 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.86461509200001, 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00059216499999999997, 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.980 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065246+0000", 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065246+0000", 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065247+0000", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2399999999999998e-07 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065284+0000", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7753000000000003e-05 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065292+0000", 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.559e-06 2024-08-06T17:53:19.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065298+0000", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2940000000000002e-06 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065459+0000", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000160686 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065479+0000", 2024-08-06T17:53:19.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9919e-05 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065798+0000", 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00031921500000000003 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.065838+0000", 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0114999999999998e-05 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.984 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:96 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'70, mlcod=16'65)", 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.123330+0000", 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.80653138999998, 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0022108200000000001, 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123330+0000", 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123330+0000", 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123330+0000", 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7899999999999999e-07 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123348+0000", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7943000000000001e-05 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123357+0000", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1519999999999998e-06 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123362+0000", 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5070000000000002e-06 2024-08-06T17:53:19.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123399+0000", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7605000000000003e-05 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.123420+0000", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0154000000000001e-05 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.125502+0000", 2024-08-06T17:53:19.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.002082038 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.125541+0000", 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8942000000000001e-05 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:112 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'82, mlcod=16'80)", 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.139101+0000", 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.79076002800002, 2024-08-06T17:53:19.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00062204899999999995, 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139101+0000", 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139101+0000", 2024-08-06T17:53:19.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139102+0000", 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5499999999999997e-07 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139107+0000", 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.434e-06 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.991 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139120+0000", 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2706e-05 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139126+0000", 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4200000000000004e-06 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139168+0000", 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1464000000000001e-05 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139181+0000", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3062e-05 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139669+0000", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000487601 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139723+0000", 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4607000000000001e-05 2024-08-06T17:53:19.993 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:114 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'84, mlcod=16'80)", 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.139769+0000", 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.79009241300002, 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000624352, 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139769+0000", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139769+0000", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139769+0000", 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.0399999999999996e-07 2024-08-06T17:53:19.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139797+0000", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7341000000000001e-05 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139807+0000", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8050000000000001e-06 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139813+0000", 2024-08-06T17:53:19.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.584e-06 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139880+0000", 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6804999999999995e-05 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.139900+0000", 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.039e-05 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.140362+0000", 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00046170100000000002 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.140393+0000", 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1121999999999998e-05 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:119 1.0 e16/14 1:30306672:devicehealth::main.db.0000000000000000:head v 16'89, mlcod=16'87)", 2024-08-06T17:53:19.998 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.142793+0000", 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.787068288, 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00062558199999999996, 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142793+0000", 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:19.999 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142793+0000", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142794+0000", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4799999999999998e-07 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142813+0000", 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9389e-05 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142822+0000", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5019999999999998e-06 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142829+0000", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2790000000000004e-06 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142867+0000", 2024-08-06T17:53:20.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8248999999999999e-05 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.142885+0000", 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.84e-05 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.143372+0000", 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00048653200000000002 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.002 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.143418+0000", 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6582999999999998e-05 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "osd_repop(client.14154.0:121 1.0 e16/14 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 16'91, mlcod=16'89)", 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:45:09.144017+0000", 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 490.78584459799998, 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0010294099999999999, 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "commit sent; apply or cleanup", 2024-08-06T17:53:20.003 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144017+0000", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144017+0000", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144017+0000", 2024-08-06T17:53:20.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5600000000000005e-07 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144022+0000", 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1170000000000003e-06 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144033+0000", 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0841e-05 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144040+0000", 2024-08-06T17:53:20.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.409e-06 2024-08-06T17:53:20.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144082+0000", 2024-08-06T17:53:20.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:20.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2391000000000001e-05 2024-08-06T17:53:20.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.144097+0000", 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5057e-05 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "commit_sent", 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.145003+0000", 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00090575499999999995 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:45:09.145046+0000", 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3183999999999999e-05 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPull(1.0 e47/46 cost 1049576)", 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:47:03.659041+0000", 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 376.27081974200001, 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00080504000000000003, 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-08-06T17:53:20.009 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659041+0000", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659041+0000", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659042+0000", 2024-08-06T17:53:20.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.34e-07 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659053+0000", 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1014e-05 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659060+0000", 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5100000000000001e-06 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659068+0000", 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2989999999999999e-06 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659110+0000", 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2555999999999997e-05 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.659846+0000", 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000736227 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.012 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPull(1.0 e47/46 cost 1049576)", 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:47:03.661244+0000", 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 376.26861667899999, 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00074509399999999996, 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661244+0000", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661244+0000", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661245+0000", 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3900000000000005e-07 2024-08-06T17:53:20.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661252+0000", 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4089999999999997e-06 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661261+0000", 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8400000000000001e-06 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661266+0000", 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4630000000000003e-06 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661307+0000", 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1698e-05 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:03.661990+0000", 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000682145 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.016 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPull(1.0 e55/54 cost 1049576)", 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:47:27.126752+0000", 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 352.80310872000001, 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00070301800000000002, 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126752+0000", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126752+0000", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126758+0000", 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3469999999999998e-06 2024-08-06T17:53:20.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126770+0000", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2471e-05 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126780+0000", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.9210000000000004e-06 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126786+0000", 2024-08-06T17:53:20.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.19e-06 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.126825+0000", 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8727999999999999e-05 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.127455+0000", 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00063036099999999996 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDPGPull(1.0 e55/54 cost 1049576)", 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:47:27.128808+0000", 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 352.80105330999999, 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00068958099999999998, 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "reached pg", 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128808+0000", 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128808+0000", 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128808+0000", 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6600000000000002e-07 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:53:20.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128816+0000", 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1890000000000001e-06 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128823+0000", 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2099999999999996e-06 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128827+0000", 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9899999999999999e-06 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.128862+0000", 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5274000000000001e-05 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:47:27.129497+0000", 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000635452 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:53:20.024 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:53:20.025 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:53:20.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:19 smithi107 ceph-mon[32381]: pgmap v457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:20.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:19 smithi086 ceph-mon[29229]: pgmap v457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:20.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:20.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:20.299 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:19 smithi117 ceph-mon[32835]: pgmap v457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:20.299 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:20.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:20.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:20.596 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:20.597 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":161,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:16.191127+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:07.203826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:20.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:20.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:20.827 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:20.827 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:20.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:21.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:20 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:20 smithi086 ceph-mon[29229]: osdmap e162: 12 total, 3 up, 11 in 2024-08-06T17:53:21.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:21.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:20 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:21.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:21.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:20 smithi107 ceph-mon[32381]: osdmap e162: 12 total, 3 up, 11 in 2024-08-06T17:53:21.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:21.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:20 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:21.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/939645616' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:21.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:20 smithi117 ceph-mon[32835]: osdmap e162: 12 total, 3 up, 11 in 2024-08-06T17:53:21.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:21.487 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:21.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:21.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:21.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:21.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:21.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:22.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:22.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:22.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:22.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:22.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T17:53:22.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:22.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:22.423 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:22 smithi086 ceph-mon[29229]: osdmap e163: 12 total, 3 up, 11 in 2024-08-06T17:53:22.423 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:22 smithi086 ceph-mon[29229]: pgmap v460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:22 smithi107 ceph-mon[32381]: osdmap e163: 12 total, 3 up, 11 in 2024-08-06T17:53:22.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:22 smithi107 ceph-mon[32381]: pgmap v460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:22.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:22 smithi117 ceph-mon[32835]: osdmap e163: 12 total, 3 up, 11 in 2024-08-06T17:53:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:22 smithi117 ceph-mon[32835]: pgmap v460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:22.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:22.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:22.842 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:22.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:23.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:23.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:23.296 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:23.296 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:23.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:23.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:23.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:23.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:53:23.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:23 smithi117 ceph-mon[32835]: pgmap v461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:23.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:23 smithi107 ceph-mon[32381]: pgmap v461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:24.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:24.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:23 smithi086 ceph-mon[29229]: pgmap v461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:24.036 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:24.036 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:24.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:24.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:24.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:24.412 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:24.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:24.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:24.589 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:24.589 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:24.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:24.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:24.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:24.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:24.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:24.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:24.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:25.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:25.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:25.303 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:53:25.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:25.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:25.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:25.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:25.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:25.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:25.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:25.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:25.800 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:25 smithi086 ceph-mon[29229]: pgmap v462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:25.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:25.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:25 smithi117 ceph-mon[32835]: pgmap v462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:25 smithi107 ceph-mon[32381]: pgmap v462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:26.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:26.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:26.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:26.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:26.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:26.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:26.557 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:26.574 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:26.575 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:26.575 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:26.575 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:26.575 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:26.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:26.592 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:26.593 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:26.593 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:26.593 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:26.593 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:26.593 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:26.594 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:26.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:26.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:26.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:27.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2038509509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:27.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2038509509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2038509509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:27.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:27.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:27.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:27.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:27.751 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:27.751 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:27.751 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:53:27.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:27.838 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:53:27.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T17:53:27.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:53:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:27 smithi107 ceph-mon[32381]: pgmap v463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:28.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:28.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:28.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:28.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:53:28.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:27 smithi086 ceph-mon[29229]: pgmap v463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:28.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:53:28.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:27 smithi117 ceph-mon[32835]: pgmap v463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:28.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:28.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:28.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:28.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:28.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:28.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:28.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:28.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:28.949 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:28.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:28.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:29.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:28 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:29.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:53:29.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:28 smithi086 ceph-mon[29229]: osdmap e164: 12 total, 3 up, 11 in 2024-08-06T17:53:29.005 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:29.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:29.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:28 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:29.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:53:29.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:28 smithi117 ceph-mon[32835]: osdmap e164: 12 total, 3 up, 11 in 2024-08-06T17:53:29.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:28 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:29.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1659587356' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:53:29.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:28 smithi107 ceph-mon[32381]: osdmap e164: 12 total, 3 up, 11 in 2024-08-06T17:53:29.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:29.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:29.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:29.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:29.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:29.641 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:29.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:29.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:29.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:29.938 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:29 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:53:29.938 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:29 smithi117 ceph-mon[32835]: pgmap v465: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:29 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:53:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:29 smithi107 ceph-mon[32381]: pgmap v465: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:30.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:29 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:53:30.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:29 smithi086 ceph-mon[29229]: pgmap v465: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:30.024 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:30.024 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:30.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:30.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:30.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:30.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:30.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:53:30.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:30.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:30.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:30.888 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:30.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:53:30.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:30.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:30.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:31.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:53:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:53:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:31.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:53:31.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:53:31.261 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:53:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:53:31.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:53:31.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:31.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:31.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:31.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:31.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:31.802 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:31.802 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:31.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:32.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:32.072 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:32.072 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:32.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:32.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:32 smithi086 ceph-mon[29229]: osdmap e165: 12 total, 3 up, 11 in 2024-08-06T17:53:32.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:32 smithi086 ceph-mon[29229]: pgmap v467: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:32.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:32.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:32.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:32 smithi107 ceph-mon[32381]: osdmap e165: 12 total, 3 up, 11 in 2024-08-06T17:53:32.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:32 smithi107 ceph-mon[32381]: pgmap v467: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:32.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:32.590 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:32.590 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":165,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:31.194523+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:07.203826+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:32.613 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:32 smithi117 ceph-mon[32835]: osdmap e165: 12 total, 3 up, 11 in 2024-08-06T17:53:32.613 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:32 smithi117 ceph-mon[32835]: pgmap v467: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 305 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:32.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:32.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:32.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:32.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:32.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:33.049 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:53:33.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:33.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:33.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:33.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:33.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:33.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1372149591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:33.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:33.554 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1372149591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:33.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1372149591' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:33.624 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:53:33.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T17:53:33.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:33.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:33.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:34.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:34.109 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:34.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:34.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:34 smithi086 ceph-mon[29229]: pgmap v468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:34.469 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:34.495 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:34 smithi117 ceph-mon[32835]: pgmap v468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:34.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:34.631 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:34 smithi107 ceph-mon[32381]: pgmap v468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:34.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:34.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:35.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:35.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:53:35.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:35.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:35 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:53:35.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:35 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:53:35.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:35.333 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:35.333 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:35.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:35.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:35 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:53:35.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:35 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:53:35.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:35.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:35.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:35.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:35 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:53:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:35 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:53:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:35.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:35.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:35.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:35.955 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:36.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:36.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:36.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:36.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:36.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:36 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:53:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:36 smithi117 ceph-mon[32835]: osdmap e166: 12 total, 3 up, 11 in 2024-08-06T17:53:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:36 smithi117 ceph-mon[32835]: pgmap v470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:36 smithi117 ceph-mon[32835]: osdmap e167: 12 total, 3 up, 11 in 2024-08-06T17:53:36.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:36.518 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:36 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:53:36.519 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:36.519 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:36 smithi107 ceph-mon[32381]: osdmap e166: 12 total, 3 up, 11 in 2024-08-06T17:53:36.519 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:36 smithi107 ceph-mon[32381]: pgmap v470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:36.519 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:36 smithi107 ceph-mon[32381]: osdmap e167: 12 total, 3 up, 11 in 2024-08-06T17:53:36.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:36.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:36.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:36 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:53:36.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1847540994' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:36.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:36 smithi086 ceph-mon[29229]: osdmap e166: 12 total, 3 up, 11 in 2024-08-06T17:53:36.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:36 smithi086 ceph-mon[29229]: pgmap v470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:36.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:36 smithi086 ceph-mon[29229]: osdmap e167: 12 total, 3 up, 11 in 2024-08-06T17:53:36.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:36.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:36.895 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:37.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:37.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:37.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:37.078 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:37.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:37.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:37.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:37.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:37.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:37.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:37.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:37.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:37 smithi107 ceph-mon[32381]: pgmap v472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:37.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:37.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:37 smithi086 ceph-mon[29229]: pgmap v472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:38.000 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:38.000 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:38.001 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:37 smithi117 ceph-mon[32835]: pgmap v472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:38.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:38.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:38.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:38.513 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:38.513 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:38.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:38.611 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:38.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:38.612 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:38.628 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:38.629 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:38.629 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:38.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:38.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:38.646 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:38.647 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:38.648 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:38.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:38.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1427430730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:38.780 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:53:38.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1427430730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:38.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:39.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:39.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1427430730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:39.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:39.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:39.446 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:53:39.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T17:53:39.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:39.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:39.753 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:39.753 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:39.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:39.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:39.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:53:39.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:39.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:39 smithi107 ceph-mon[32381]: pgmap v473: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:40.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:40.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:40.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:40.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:39 smithi086 ceph-mon[29229]: pgmap v473: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:40.120 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:40.120 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:39 smithi117 ceph-mon[32835]: pgmap v473: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:40.127 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:40.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:40.160 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:40.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:40.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:40.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:40.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:40.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:40.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:41.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:41.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:41.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:41.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:41.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:41 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:53:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:41 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:53:41.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:41 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:53:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:41 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:53:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:41 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:53:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:41 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:53:41.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:42.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:42.057 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:42.057 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":167,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:36.196083+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:52:20.168567+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:42.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:53:42.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:42.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:42.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:42.511 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:42.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:42.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:42.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:42.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:42 smithi107 ceph-mon[32381]: pgmap v474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4006993372' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:42.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:42.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:42 smithi117 ceph-mon[32835]: pgmap v474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:42.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4006993372' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:42 smithi086 ceph-mon[29229]: pgmap v474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4006993372' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:42.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:42.991 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:42.991 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:43.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:43.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:43.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:43.386 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:43.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:43.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:43.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:43.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:43.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:43.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:43.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:43 smithi086 ceph-mon[29229]: pgmap v475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:43.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:43 smithi107 ceph-mon[32381]: pgmap v475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:44.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:43 smithi117 ceph-mon[32835]: pgmap v475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:44.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:44.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:44.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:44.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:44.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:44.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:44.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:44.754 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:53:44.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:44.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:45.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:45.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:53:45.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:45.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:45.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:45.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:45.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:45.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:45.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:45.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:45 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:45.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:45.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:45 smithi086 ceph-mon[29229]: osdmap e168: 12 total, 3 up, 11 in 2024-08-06T17:53:45.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:45 smithi086 ceph-mon[29229]: pgmap v477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:45.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:45.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:45.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:45 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:45.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:45.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:45 smithi107 ceph-mon[32381]: osdmap e168: 12 total, 3 up, 11 in 2024-08-06T17:53:45.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:45 smithi107 ceph-mon[32381]: pgmap v477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:46.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:45 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2303415007' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:53:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:45 smithi117 ceph-mon[32835]: osdmap e168: 12 total, 3 up, 11 in 2024-08-06T17:53:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:45 smithi117 ceph-mon[32835]: pgmap v477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:46.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:46.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:46.351 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:46.351 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:46.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:46.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:46.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:46.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:46.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:46.727 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:46.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:46.939 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:46.939 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:46.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:47.038 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:47.039 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:47.039 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:47.054 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:47.054 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:47.054 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:47.055 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:47.055 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:47.074 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:47.075 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:47.075 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:47.075 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:47.075 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:47.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:47.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:47.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:47.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:47.294 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:47 smithi117 ceph-mon[32835]: osdmap e169: 12 total, 3 up, 11 in 2024-08-06T17:53:47.294 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4196546813' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:47.294 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:47 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:53:47.294 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:47 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:53:47.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:47 smithi107 ceph-mon[32381]: osdmap e169: 12 total, 3 up, 11 in 2024-08-06T17:53:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4196546813' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:47 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:53:47.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:47 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:53:47.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:47.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:47.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:47.538 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:47 smithi086 ceph-mon[29229]: osdmap e169: 12 total, 3 up, 11 in 2024-08-06T17:53:47.539 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4196546813' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:47.539 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:47 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:53:47.539 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:47 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:53:47.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:47.711 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:47.712 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:47.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:47.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:53:47.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:48.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:48.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:48.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:48.365 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:48.365 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.366 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.366 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.366 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:48 smithi107 ceph-mon[32381]: osdmap e170: 12 total, 3 up, 11 in 2024-08-06T17:53:48.366 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:48 smithi107 ceph-mon[32381]: pgmap v480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:48.447 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:48.448 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:48.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:48 smithi086 ceph-mon[29229]: osdmap e170: 12 total, 3 up, 11 in 2024-08-06T17:53:48.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:48 smithi086 ceph-mon[29229]: pgmap v480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:48.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:48.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:48 smithi117 ceph-mon[32835]: osdmap e170: 12 total, 3 up, 11 in 2024-08-06T17:53:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:48 smithi117 ceph-mon[32835]: pgmap v480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:48.630 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:48.630 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:48.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:48.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:49.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:49.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:49.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:49.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:49.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:49.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:49.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:49.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:49.493 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:53:49.493 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":170,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:47.206210+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:53:49.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:49.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4227897310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:49.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:49.695 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4227897310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:49.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:53:49.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:49.830 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4227897310' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:53:49.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:50.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:50.099 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:50.099 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:50.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:50.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:50.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:50.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:50.638 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:50 smithi117 ceph-mon[32835]: pgmap v481: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:50.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:50.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:50 smithi107 ceph-mon[32381]: pgmap v481: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:50.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:50 smithi086 ceph-mon[29229]: pgmap v481: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:50.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:50.897 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:50.928 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:53:50.928 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:53:51.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:51.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:51.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:51.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:51.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:51.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:51.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:51.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:51.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:51.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:51.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:51.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:52.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:52.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:52.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T17:53:52.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:52.388 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:52 smithi107 ceph-mon[32381]: osdmap e171: 12 total, 3 up, 11 in 2024-08-06T17:53:52.389 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:52 smithi107 ceph-mon[32381]: pgmap v483: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:52.389 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:52.389 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:52.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:52.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:52 smithi086 ceph-mon[29229]: osdmap e171: 12 total, 3 up, 11 in 2024-08-06T17:53:52.520 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:52 smithi086 ceph-mon[29229]: pgmap v483: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:52.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:52 smithi117 ceph-mon[32835]: osdmap e171: 12 total, 3 up, 11 in 2024-08-06T17:53:52.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:52 smithi117 ceph-mon[32835]: pgmap v483: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:52.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:52.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:52.826 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:52.826 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:52.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:53.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:53.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:53.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:53.406 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:53.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:53.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:53.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:53:53.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:53.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:53.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:53.801 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:53.801 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:53.802 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:53 smithi086 ceph-mon[29229]: pgmap v484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:53.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:53.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:54.008 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:53 smithi107 ceph-mon[32381]: pgmap v484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:54.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:53 smithi117 ceph-mon[32835]: pgmap v484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:54.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:54.277 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:54.278 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:54.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:54.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:54.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:54.753 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:53:54.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:54.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:55.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:55.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:55.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:53:55.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:55.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:55.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:55.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:55.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:55.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:53:55.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:55.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:55.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:55.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:55 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:55 smithi086 ceph-mon[29229]: osdmap e172: 12 total, 3 up, 11 in 2024-08-06T17:53:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:55 smithi086 ceph-mon[29229]: pgmap v486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:55.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:55 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:55 smithi107 ceph-mon[32381]: osdmap e172: 12 total, 3 up, 11 in 2024-08-06T17:53:56.016 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:55 smithi107 ceph-mon[32381]: pgmap v486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:55 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:53:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/947189141' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:53:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:55 smithi117 ceph-mon[32835]: osdmap e172: 12 total, 3 up, 11 in 2024-08-06T17:53:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:55 smithi117 ceph-mon[32835]: pgmap v486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:56.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:56.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:56.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:56.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:56.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:56.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:56.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:56.506 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:53:56.507 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:53:56.508 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:53:56.508 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:53:56.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:56.538 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:53:56.538 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:53:56.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:53:56.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:53:56.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:53:56.558 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:53:56.558 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:53:56.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:53:56.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:53:56.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:53:56.559 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:53:56.559 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:53:56.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:56.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:56.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:57.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:57.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:57.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:57.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:53:57.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:57.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:57.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:57.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:57 smithi086 ceph-mon[29229]: osdmap e173: 12 total, 3 up, 11 in 2024-08-06T17:53:57.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/843077917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:57.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:57.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:57 smithi107 ceph-mon[32381]: osdmap e173: 12 total, 3 up, 11 in 2024-08-06T17:53:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/843077917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:53:57.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:57 smithi117 ceph-mon[32835]: osdmap e173: 12 total, 3 up, 11 in 2024-08-06T17:53:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/843077917' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:53:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:57.702 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:53:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:53:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T17:53:57.702 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T17:53:57.702 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:53:57.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:57.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:57.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:57.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:53:57.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:58.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:58.208 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:58.208 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:53:58.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:58.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:53:58.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:58.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:58 smithi086 ceph-mon[29229]: pgmap v488: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:58.418 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:58.419 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:58.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:58.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:58 smithi107 ceph-mon[32381]: pgmap v488: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:58.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:53:58.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:53:58.578 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:53:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:53:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:58 smithi117 ceph-mon[32835]: pgmap v488: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:58.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:58.692 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:53:58.692 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:53:58.693 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:53:58.693 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:53:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:53:58.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:58.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:53:59.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T17:53:59.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:53:59.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:53:59.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:53:59.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:59.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:53:59.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:53:59.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:53:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:53:59 smithi107 ceph-mon[32381]: pgmap v489: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:53:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:53:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:53:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:00.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:00.012 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:00.055 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:53:59 smithi086 ceph-mon[29229]: pgmap v489: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:53:59 smithi117 ceph-mon[32835]: pgmap v489: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:00.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:00.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:00.201 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":173,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:53:56.201133+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:00.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:00.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:00.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:00.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:54:00.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:00.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:00.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3502604102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:00.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3502604102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:00.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3502604102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:01.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:01.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:01.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:01.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:01.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:01.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:01.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:02.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:02.022 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:02.023 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:02.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:02.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:02 smithi086 ceph-mon[29229]: pgmap v490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:02.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:02.313 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:02 smithi117 ceph-mon[32835]: pgmap v490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:02.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:02 smithi107 ceph-mon[32381]: pgmap v490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:02.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:02.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:02.480 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:02.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:02.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:02.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:02.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:03.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:03.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:03.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:03.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:03.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:03.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:03.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:03.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:03.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:03.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:03.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:03.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:03.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:03 smithi107 ceph-mon[32381]: pgmap v491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:04.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:04.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:03 smithi086 ceph-mon[29229]: pgmap v491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:04.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:03 smithi117 ceph-mon[32835]: pgmap v491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:04.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:04.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:04.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:04.561 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:04.561 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:04.561 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:04.570 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:04.571 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:04.571 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:04.571 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:04.571 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:04.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:04.589 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:04.590 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:04.590 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:04.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:04.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:04.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:04.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:04.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:04.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:04.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:04.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3541885169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:04.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:04.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3541885169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:04.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:05.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:05.172 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:54:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T17:54:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T17:54:05.172 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:167,from:0,reservation_nonce:0)", 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.344123663000001, 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208364, 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:54:05.173 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:54:05.174 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7299999999999995e-07 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123197+0000", 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7089999999999998e-06 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123207+0000", 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.175 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123212+0000", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6380000000000001e-06 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123253+0000", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0308000000000001e-05 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123272+0000", 2024-08-06T17:54:05.176 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9191e-05 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123402+0000", 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013016300000000001 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T17:54:05.177 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.343435665000001, 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8758999999999996e-05, 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:54:05.178 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7800000000000002e-07 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123885+0000", 2024-08-06T17:54:05.179 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9720000000000001e-06 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123892+0000", 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2119999999999997e-06 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123896+0000", 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8580000000000002e-06 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:54:05.180 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123933+0000", 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6906000000000002e-05 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123941+0000", 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8709999999999997e-06 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123960+0000", 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9462000000000001e-05 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:54:05.181 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:54:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:54:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:54:05.182 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:54:05.182 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:54:05.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:05.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:05.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3541885169' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:05.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:05.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:05.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:05.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:05.670 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:05.670 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:05.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:05.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T17:54:05.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:05 smithi107 ceph-mon[32381]: pgmap v492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:05.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:05.938 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:05.938 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:05.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:05.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:05 smithi086 ceph-mon[29229]: pgmap v492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:06.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:06.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:06.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:06.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:06.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:05 smithi117 ceph-mon[32835]: pgmap v492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:06.118 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:06.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:06.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:06.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:06.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:06.606 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:06.607 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:06.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:06.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:06.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:06.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:06.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:06.997 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:07.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:07.203 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:54:07.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:07.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:07.417 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:07.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:07.417 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:07.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:07.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:07.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:07.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:07.754 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:07.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:07.856 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:07.857 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:07.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:07.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:07.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:08.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:08.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:08.316 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:08 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:08.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:08.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:08 smithi107 ceph-mon[32381]: osdmap e174: 12 total, 3 up, 11 in 2024-08-06T17:54:08.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:08 smithi107 ceph-mon[32381]: pgmap v494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:08.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:08.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:08.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T17:54:08.435 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:08 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:08.435 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:08.436 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:08 smithi086 ceph-mon[29229]: osdmap e174: 12 total, 3 up, 11 in 2024-08-06T17:54:08.436 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:08 smithi086 ceph-mon[29229]: pgmap v494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:08.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:08.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:08 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:08.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/672038325' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:08.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:08 smithi117 ceph-mon[32835]: osdmap e174: 12 total, 3 up, 11 in 2024-08-06T17:54:08.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:08 smithi117 ceph-mon[32835]: pgmap v494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:08.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:08.611 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:08.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:08.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:54:08.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:09.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:09.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:09.279 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=58,logm=2} crc {auth=1551532477,config=507150784,health=3635590522,kv=3848604320,logm=1282435356}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1987697682}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770270086}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1395137443}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164361894}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628948375}) 2024-08-06T17:54:09.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824638470}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T17:54:09.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546165370}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174291596}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922676}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067476590}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1313832907}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991590651}) 2024-08-06T17:54:09.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775513737}) 2024-08-06T17:54:09.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439385315}) 2024-08-06T17:54:09.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826918013}) 2024-08-06T17:54:09.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073884263}) 2024-08-06T17:54:09.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=6} crc {logm=518499660,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1065455420,mon_config_key=3588673942}) 2024-08-06T17:54:09.370 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4014830010}) 2024-08-06T17:54:09.371 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=879103390}) 2024-08-06T17:54:09.371 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=709066307,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3827319726,osdmap=4031520870}) 2024-08-06T17:54:09.371 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=499519712}) 2024-08-06T17:54:09.371 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3754220019}) 2024-08-06T17:54:09.371 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=2563344442}) 2024-08-06T17:54:09.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:09.433 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=58,logm=2} crc {auth=1551532477,config=507150784,health=3635590522,kv=3848604320,logm=1282435356}) 2024-08-06T17:54:09.433 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1987697682}) 2024-08-06T17:54:09.433 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770270086}) 2024-08-06T17:54:09.433 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1395137443}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164361894}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628948375}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824638470}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T17:54:09.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546165370}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174291596}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922676}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067476590}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1313832907}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991590651}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775513737}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439385315}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826918013}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073884263}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=6} crc {logm=518499660,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1065455420,mon_config_key=3588673942}) 2024-08-06T17:54:09.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4014830010}) 2024-08-06T17:54:09.436 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=879103390}) 2024-08-06T17:54:09.436 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=709066307,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3827319726,osdmap=4031520870}) 2024-08-06T17:54:09.436 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=499519712}) 2024-08-06T17:54:09.436 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3754220019}) 2024-08-06T17:54:09.436 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=2563344442}) 2024-08-06T17:54:09.436 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:09.437 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":174,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:07.203160+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:14.854160+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:09.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=58,logm=2} crc {auth=1551532477,config=507150784,health=3635590522,kv=3848604320,logm=1282435356}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1987697682}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3770270086}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1395137443}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=164361894}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628948375}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2824638470}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T17:54:09.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2546165370}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1174291596}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1089922676}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1067476590}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1313832907}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=991590651}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1775513737}) 2024-08-06T17:54:09.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1439385315}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2826918013}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073884263}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=53,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=6} crc {logm=518499660,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1065455420,mon_config_key=3588673942}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4014830010}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=879103390}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=73,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=9} crc {mon_config_key=709066307,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3827319726,osdmap=4031520870}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=499519712}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3754220019}) 2024-08-06T17:54:09.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=18} crc {osdmap=2563344442}) 2024-08-06T17:54:09.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:09.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:09.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:09.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:09.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:54:09.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:10.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:10.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:10.319 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:10.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:10.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3229906748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:10.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:10 smithi117 ceph-mon[32835]: pgmap v495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:10.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:10.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3229906748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:10.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:10 smithi107 ceph-mon[32381]: pgmap v495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:10.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:10.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:10.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:10.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3229906748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:10.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:10 smithi086 ceph-mon[29229]: pgmap v495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:10.647 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:10.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:10.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:10.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:10.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:10.892 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:10.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T17:54:10.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:11.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:11.179 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:11.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:11.328 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:11.328 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:11.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:11.345 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:11.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:11.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:11.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:11.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:11.362 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:11.362 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:11.362 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:11.362 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:11.363 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:11.363 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:11.363 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:11.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:11.428 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:11.428 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:11.437 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:11.437 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:11.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:11.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:11.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:11.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:12.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:12.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:12.285 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:12.285 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:12.285 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: osdmap e175: 12 total, 3 up, 11 in 2024-08-06T17:54:12.285 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1644867570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:12.286 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.286 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.286 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.286 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:12 smithi107 ceph-mon[32381]: pgmap v497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:12.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: osdmap e175: 12 total, 3 up, 11 in 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1644867570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:12 smithi086 ceph-mon[29229]: pgmap v497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:12.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:12.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: osdmap e175: 12 total, 3 up, 11 in 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1644867570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:12.608 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:12 smithi117 ceph-mon[32835]: pgmap v497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:12.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:12.738 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:12.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:12.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:12.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:12.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:13.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:13.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:13.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:13.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:13.357 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:13.382 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:13.382 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:13.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:13.412 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:13.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T17:54:13.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:13.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:13.703 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:13.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:13 smithi086 ceph-mon[29229]: pgmap v498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:13 smithi117 ceph-mon[32835]: pgmap v498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:13.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:13.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:13 smithi107 ceph-mon[32381]: pgmap v498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:13.954 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:13.954 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:14.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:14.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:14.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:14.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:14.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:14.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:14.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:14.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:14.758 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:54:14.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:14.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:14.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:15.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:15.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:15.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:15.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:15.326 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:15.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:54:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:15.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:15.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:15.542 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:15.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:15.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:15.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:15 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:15.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:15.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:15 smithi107 ceph-mon[32381]: osdmap e176: 12 total, 3 up, 11 in 2024-08-06T17:54:15.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:15 smithi107 ceph-mon[32381]: pgmap v500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:15.913 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:15.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T17:54:15.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:15.935 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:15 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:15.935 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:15.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:15 smithi086 ceph-mon[29229]: osdmap e176: 12 total, 3 up, 11 in 2024-08-06T17:54:15.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:15 smithi086 ceph-mon[29229]: pgmap v500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:16.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:15 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2919740296' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:15 smithi117 ceph-mon[32835]: osdmap e176: 12 total, 3 up, 11 in 2024-08-06T17:54:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:15 smithi117 ceph-mon[32835]: pgmap v500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:16.162 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:16.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:16.244 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:16.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:16.346 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:16.346 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:16.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:16.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:16.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:16.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:16.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:16.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:16.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:17.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:17.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:17 smithi086 ceph-mon[29229]: osdmap e177: 12 total, 3 up, 11 in 2024-08-06T17:54:17.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:17.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:17.407 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:17 smithi107 ceph-mon[32381]: osdmap e177: 12 total, 3 up, 11 in 2024-08-06T17:54:17.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:54:17.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:17.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:17.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:17.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:17 smithi117 ceph-mon[32835]: osdmap e177: 12 total, 3 up, 11 in 2024-08-06T17:54:17.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:17.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:17.821 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:17.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:17.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:17.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:18.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:18.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:18.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:18.416 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:18 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:54:18.416 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:18 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:54:18.416 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:18 smithi107 ceph-mon[32381]: pgmap v502: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:18.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:18.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T17:54:18.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:18.475 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:18.475 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:18.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:18 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:54:18.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:18 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:54:18.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:18 smithi086 ceph-mon[29229]: pgmap v502: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:18.490 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:18.531 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:18.532 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":178,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:18.209579+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:18.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:18 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:54:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:18 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:54:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:18 smithi117 ceph-mon[32835]: pgmap v502: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:18.696 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:18.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:18.738 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:18.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:18.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:18.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:18.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:18.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:19.093 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:19.094 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:19.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:19.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:19.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:19 smithi086 ceph-mon[29229]: osdmap e178: 12 total, 3 up, 11 in 2024-08-06T17:54:19.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3736802432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:19.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:19.424 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:19 smithi107 ceph-mon[32381]: osdmap e178: 12 total, 3 up, 11 in 2024-08-06T17:54:19.425 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3736802432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:19.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:19.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:19 smithi117 ceph-mon[32835]: osdmap e178: 12 total, 3 up, 11 in 2024-08-06T17:54:19.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3736802432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:19.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:19.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:19.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:19.690 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:19.690 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:19.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:19.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:19.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:19.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:19.830 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:19.830 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:19.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:19.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:19.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:19.847 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:19.848 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:19.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:19.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:19.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:19.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:19.849 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:19.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:20.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:20.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:20.300 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:20.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:20.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:20.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:20 smithi086 ceph-mon[29229]: pgmap v504: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:20.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2800884340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:20.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.307 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:20.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:20.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:20 smithi107 ceph-mon[32381]: pgmap v504: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:20.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2800884340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:20.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:20.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:20 smithi117 ceph-mon[32835]: pgmap v504: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:20.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2800884340' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:20.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:20.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:20.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:20.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:20.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T17:54:20.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:21.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:21.177 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:21.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:21.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:21.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:21.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:21.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:21.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:21 smithi086 ceph-mon[29229]: osdmap e179: 12 total, 3 up, 11 in 2024-08-06T17:54:21.525 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:21.526 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:21.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:21.539 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:21.539 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:21.540 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:21.540 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:21 smithi107 ceph-mon[32381]: osdmap e179: 12 total, 3 up, 11 in 2024-08-06T17:54:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:21 smithi117 ceph-mon[32835]: osdmap e179: 12 total, 3 up, 11 in 2024-08-06T17:54:21.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:21.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:21.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:22.017 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:22.018 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:22.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:22.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:22.224 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:54:22.268 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:22 smithi107 ceph-mon[32381]: pgmap v506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:22.269 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:22.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:22.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:22 smithi086 ceph-mon[29229]: pgmap v506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:22.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:22.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:22.481 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:22 smithi117 ceph-mon[32835]: pgmap v506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:22.481 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:22.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:22.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:22.779 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:54:22.779 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:54:22.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:22.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:23.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:23.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:23.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:23 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:23.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:23.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:23 smithi107 ceph-mon[32381]: osdmap e180: 12 total, 3 up, 11 in 2024-08-06T17:54:23.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T17:54:23.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:23.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:23.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:23.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:54:23.550 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:23.550 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:23.550 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:23 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:23.550 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:23.551 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:23 smithi086 ceph-mon[29229]: osdmap e180: 12 total, 3 up, 11 in 2024-08-06T17:54:23.581 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:23.584 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:23 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1121774256' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:23 smithi117 ceph-mon[32835]: osdmap e180: 12 total, 3 up, 11 in 2024-08-06T17:54:23.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:23.674 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:23.674 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:23.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:23.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:24.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:24.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:24.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:24.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:24.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:24.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:24 smithi107 ceph-mon[32381]: pgmap v508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:24.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:24.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:24 smithi117 ceph-mon[32835]: pgmap v508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:24 smithi086 ceph-mon[29229]: pgmap v508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:24.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:24.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:24.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:24.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:24.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:54:24.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:25.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:25.144 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:54:25.144 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:54:25.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:25.275 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:25.276 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:25.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:25.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:25.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:25.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:25.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:25 smithi086 ceph-mon[29229]: pgmap v509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:25.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T17:54:25.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:25 smithi107 ceph-mon[32381]: pgmap v509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:26.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:26.052 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:26.052 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":180,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:22.221335+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:26.057 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:26.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:25 smithi117 ceph-mon[32835]: pgmap v509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 309 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:26.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:26.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:26.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:26.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:26.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:26.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:26.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:27.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2174816378' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:27.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:26 smithi117 ceph-mon[32835]: osdmap e181: 12 total, 3 up, 11 in 2024-08-06T17:54:27.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2174816378' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:27.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:26 smithi086 ceph-mon[29229]: osdmap e181: 12 total, 3 up, 11 in 2024-08-06T17:54:27.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:27.129 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2174816378' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:27.129 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:26 smithi107 ceph-mon[32381]: osdmap e181: 12 total, 3 up, 11 in 2024-08-06T17:54:27.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:27.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:27.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:27.631 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:54:27.632 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:54:27.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:27.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:27.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:27.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:27.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:27.809 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:27.809 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:27.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:27.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:27.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:27.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:27.827 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:27.828 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:27.828 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:27.828 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:27.829 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:27.829 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:27.829 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:27.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:27.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:27 smithi086 ceph-mon[29229]: pgmap v511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:27.918 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:27.918 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:28.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:28.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:27 smithi117 ceph-mon[32835]: pgmap v511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:28.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:27 smithi107 ceph-mon[32381]: pgmap v511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:28.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:28.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T17:54:28.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:28.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:28.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:28.467 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:28.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:28.548 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:28.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:28.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:28.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:28.571 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:28.571 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:28.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:28.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:28.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/899179271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:28.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/899179271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/899179271' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:28.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:29.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:29.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:29.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:29.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:29.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:29.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:29.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:29 smithi107 ceph-mon[32381]: pgmap v512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:29.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:30.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:29 smithi086 ceph-mon[29229]: pgmap v512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:30.013 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:30.013 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:30.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:29 smithi117 ceph-mon[32835]: pgmap v512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:30.131 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:30.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:30.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:30.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:30.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:54:30.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:30.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:30.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:30.743 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:30.754 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:30.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T17:54:30.768 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:30.768 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:30.773 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:54:30.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:30.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:30.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:30.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:30 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:54:31.015 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:31.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:31.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:30 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:54:31.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:31.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:31.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:30 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:54:31.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:31.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:31.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:31.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:31.643 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:31.659 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:31.659 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:31.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:31.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: osdmap e182: 12 total, 3 up, 11 in 2024-08-06T17:54:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: osdmap e183: 12 total, 3 up, 11 in 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:31.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:31 smithi107 ceph-mon[32381]: pgmap v515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:32.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:32.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: osdmap e182: 12 total, 3 up, 11 in 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: osdmap e183: 12 total, 3 up, 11 in 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:31 smithi117 ceph-mon[32835]: pgmap v515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:32.089 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:32.089 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3443786809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: osdmap e182: 12 total, 3 up, 11 in 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: osdmap e183: 12 total, 3 up, 11 in 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:54:32.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:31 smithi086 ceph-mon[29229]: pgmap v515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:32.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:32.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:32.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:32.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:32.586 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:32.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:32.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:32.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:33.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:33.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T17:54:33.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:33.248 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:33.248 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:33.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:33.435 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:54:33.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:33.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:33.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:33.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:33.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:33.813 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:33 smithi107 ceph-mon[32381]: pgmap v516: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:33.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:33 smithi086 ceph-mon[29229]: pgmap v516: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:33.838 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:33.838 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:33.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:33.918 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:33 smithi117 ceph-mon[32835]: pgmap v516: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:34.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:34.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:54:34.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:34.251 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:34.251 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":183,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:31.211184+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:34.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:34.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:34.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:34.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:34.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:34.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:34.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2573065293' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:34.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:34.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2573065293' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:34.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:34.991 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2573065293' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:35.035 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:35.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:35.193 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:35.194 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:35.194 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:35.212 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:35.212 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:35.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:35.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:35.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:35.229 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:35.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:35.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:35.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:35.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:35.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:35.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:35.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:35.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:35.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:35.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:35.624 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:35.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T17:54:35.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:35.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/935190816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:35 smithi086 ceph-mon[29229]: pgmap v517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:35.881 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:35.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:35.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:35.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/935190816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:35 smithi107 ceph-mon[32381]: pgmap v517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:36.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/935190816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:36.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:36.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:35 smithi117 ceph-mon[32835]: pgmap v517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:36.065 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:36.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:36.093 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:36.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:36.224 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:36.224 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:36.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:36.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:36.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:36.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:36.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:36.887 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:36.887 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:36.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:37.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:37.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:37.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:37.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:37.155 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:37.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:37.377 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:37.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:37.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:37.659 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:37.659 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:37.659 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:37.659 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:37.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:37.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:37.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:37.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:37 smithi117 ceph-mon[32835]: pgmap v518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:37.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:37.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T17:54:38.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:38.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:38.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:37 smithi086 ceph-mon[29229]: pgmap v518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:38.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:54:38.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:37 smithi107 ceph-mon[32381]: pgmap v518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:38.265 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:38.306 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:38.310 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:38.310 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:38.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:38.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:38.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:38.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:38.772 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:54:38.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:38.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:38.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:39.013 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:39.019 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:54:39.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:39.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:39.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:39.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:39.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:39.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:39.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:39 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:39.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:39.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:39 smithi117 ceph-mon[32835]: osdmap e184: 12 total, 3 up, 11 in 2024-08-06T17:54:39.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:39 smithi117 ceph-mon[32835]: pgmap v520: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:39 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:39 smithi086 ceph-mon[29229]: osdmap e184: 12 total, 3 up, 11 in 2024-08-06T17:54:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:39 smithi086 ceph-mon[29229]: pgmap v520: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:39.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:39 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3408038209' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:54:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:39 smithi107 ceph-mon[32381]: osdmap e184: 12 total, 3 up, 11 in 2024-08-06T17:54:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:39 smithi107 ceph-mon[32381]: pgmap v520: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:39.972 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:40.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:40.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:40.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:40.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:40.531 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:40.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T17:54:40.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:40.679 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:40.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:40.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:40.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:40.823 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:40.864 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:40.864 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":184,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:38.767828+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:40 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:54:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:40 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:54:40.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:40 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:54:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:40 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:54:41.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:41.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:40 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:54:41.037 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:40 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:54:41.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:41.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:41.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:41.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:41.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:41.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:41.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:41.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:41.831 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1358177106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:41.831 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:41 smithi117 ceph-mon[32835]: osdmap e185: 12 total, 3 up, 11 in 2024-08-06T17:54:41.831 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:41 smithi117 ceph-mon[32835]: pgmap v522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:41.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:42.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:42.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1358177106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:42.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:41 smithi107 ceph-mon[32381]: osdmap e185: 12 total, 3 up, 11 in 2024-08-06T17:54:42.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:41 smithi107 ceph-mon[32381]: pgmap v522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1358177106' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:41 smithi086 ceph-mon[29229]: osdmap e185: 12 total, 3 up, 11 in 2024-08-06T17:54:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:41 smithi086 ceph-mon[29229]: pgmap v522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:42.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:54:42.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:42.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:42.342 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:42.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:42.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:42.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:42.748 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:42.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:42.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:42.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:42.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T17:54:43.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:43.032 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:43.032 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:43.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:43.089 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:43.089 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:43.106 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:43.107 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:43.107 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:43.107 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:43.107 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:43.124 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:43.125 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:43.126 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:43.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:43.191 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:43.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:43.297 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/236620312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:43.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/236620312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:43.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:43.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:43.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:43.498 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/236620312' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:43.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:43.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:43.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:43.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:44.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:44.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:44.246 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:44.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:44.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:44.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:44 smithi117 ceph-mon[32835]: pgmap v523: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:44.492 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.492 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.492 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.492 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:44 smithi107 ceph-mon[32381]: pgmap v523: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:44.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:44.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:44.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:44 smithi086 ceph-mon[29229]: pgmap v523: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:44.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:44.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:44.790 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:44.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:44.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:44.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:45.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:45.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:45.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:45.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:45.202 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:45.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:45.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:45.413 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:45.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T17:54:45.420 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:54:45.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:45.551 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:45.605 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:45.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:54:45.716 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:45.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:45.862 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:45.862 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:45.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:46.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:46.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:46.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:54:46.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:46.437 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:46.437 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:46.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:46.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:46.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:46 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:46 smithi107 ceph-mon[32381]: osdmap e186: 12 total, 3 up, 11 in 2024-08-06T17:54:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:46 smithi107 ceph-mon[32381]: pgmap v525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:46 smithi107 ceph-mon[32381]: osdmap e187: 12 total, 3 up, 11 in 2024-08-06T17:54:46.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:46 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:46.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:46.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:46 smithi086 ceph-mon[29229]: osdmap e186: 12 total, 3 up, 11 in 2024-08-06T17:54:46.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:46 smithi086 ceph-mon[29229]: pgmap v525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:46.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:46 smithi086 ceph-mon[29229]: osdmap e187: 12 total, 3 up, 11 in 2024-08-06T17:54:46.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:46 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:54:46.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2611090424' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:54:46.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:46 smithi117 ceph-mon[32835]: osdmap e186: 12 total, 3 up, 11 in 2024-08-06T17:54:46.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:46 smithi117 ceph-mon[32835]: pgmap v525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:46.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:46 smithi117 ceph-mon[32835]: osdmap e187: 12 total, 3 up, 11 in 2024-08-06T17:54:46.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:46.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:47.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:47.178 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:47.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:47.259 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:47.260 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:47.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:47.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:47.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:47.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:47.681 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:47.681 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":187,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:46.215698+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:47.732 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:47.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T17:54:47.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:47 smithi117 ceph-mon[32835]: pgmap v527: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1725923592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:47.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:47.977 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:48.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:48.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:47 smithi086 ceph-mon[29229]: pgmap v527: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:48.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1725923592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:48.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:47 smithi107 ceph-mon[32381]: pgmap v527: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:48.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1725923592' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:48.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:48.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:48.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:48.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:48.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:48.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:48.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:49.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:54:49.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:49.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:49.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:49.426 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:49.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:49.599 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:49.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:49.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:49 smithi117 ceph-mon[32835]: pgmap v528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:49.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:49.902 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:49.902 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:49.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:49 smithi086 ceph-mon[29229]: pgmap v528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:50.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:50.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:50.190 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:50.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T17:54:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:49 smithi107 ceph-mon[32381]: pgmap v528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:50.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:50.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:50.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:50.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:50.471 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:50.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:50.488 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:50.488 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:50.489 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:50.489 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:50.489 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:50.506 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:50.506 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:50.507 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:50.507 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:50.507 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:50.507 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:50.507 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:50.508 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:50.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:50.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:50.757 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:50.757 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:50.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1165821367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:50.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1165821367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:51.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:51.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1165821367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:51.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:51.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:51.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:51.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:51.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:51.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:51.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:51.795 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:51.796 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:51.895 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:52.019 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.020 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.020 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:51 smithi117 ceph-mon[32835]: pgmap v529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:52.020 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:52.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:52.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:51 smithi086 ceph-mon[29229]: pgmap v529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:52.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:52.186 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.186 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:52.186 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:51 smithi107 ceph-mon[32381]: pgmap v529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:52.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:52.322 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:52.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:52.454 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:52.455 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:52.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:52.617 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:52.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T17:54:52.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:52.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:52.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:52.863 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:52.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:53.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:53.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:53.227 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:54:53.245 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:53.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:53.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:53.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:53.517 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:54:53.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:53.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:53.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:54:53.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:53.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:53.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:54.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:54.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:54.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:54.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:54.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:54.299 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:54.340 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:54.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:54.452 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:54 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:54 smithi086 ceph-mon[29229]: osdmap e188: 12 total, 3 up, 11 in 2024-08-06T17:54:54.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:54 smithi086 ceph-mon[29229]: pgmap v531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:54.460 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:54:54.461 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":188,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:54:53.220296+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:34.506886+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:54:54.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:54 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:54 smithi107 ceph-mon[32381]: osdmap e188: 12 total, 3 up, 11 in 2024-08-06T17:54:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:54 smithi107 ceph-mon[32381]: pgmap v531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:54.563 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:54.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:54 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:54:54.587 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/556590382' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:54:54.587 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:54 smithi117 ceph-mon[32835]: osdmap e188: 12 total, 3 up, 11 in 2024-08-06T17:54:54.587 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:54 smithi117 ceph-mon[32835]: pgmap v531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:54.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:54.706 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:54.707 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:54.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:54.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:54:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:55.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T17:54:55.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:55.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:55.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3087926458' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:55.318 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:55.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:55.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3087926458' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:55.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:55.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:55.484 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:55.506 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3087926458' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:54:55.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:55.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:55.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:55.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:55.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:56.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:56.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:56.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:56.297 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:56 smithi086 ceph-mon[29229]: pgmap v532: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:56.297 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:56 smithi086 ceph-mon[29229]: osdmap e189: 12 total, 3 up, 11 in 2024-08-06T17:54:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:56 smithi117 ceph-mon[32835]: pgmap v532: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:56 smithi117 ceph-mon[32835]: osdmap e189: 12 total, 3 up, 11 in 2024-08-06T17:54:56.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:56.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:56.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:56.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:56 smithi107 ceph-mon[32381]: pgmap v532: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:56.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:56 smithi107 ceph-mon[32381]: osdmap e189: 12 total, 3 up, 11 in 2024-08-06T17:54:56.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:54:56.770 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:56.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:56.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:56.959 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:56.959 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:57.059 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:54:57.059 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:54:57.059 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:54:57.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:57.075 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:54:57.076 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:54:57.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:54:57.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:54:57.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:54:57.099 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:54:57.099 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:54:57.099 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:54:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:54:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:54:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:54:57.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:54:57.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:57.290 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/453851663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:57.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T17:54:57.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:57.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/453851663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:57.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:57.561 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:57.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:54:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/453851663' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:54:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:57.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:57.787 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:54:57.787 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:54:57.857 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:57.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:54:57.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:58.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:58.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:58.297 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:58.297 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:58.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:58 smithi086 ceph-mon[29229]: pgmap v534: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:58.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:54:58.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:54:58.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:58.559 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:58.559 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:58.580 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.580 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.580 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.580 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:58 smithi117 ceph-mon[32835]: pgmap v534: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:58.610 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:54:58.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:54:58.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:54:58.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:58 smithi107 ceph-mon[32381]: pgmap v534: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:58.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:58.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:54:59.033 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:54:59.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:54:59.085 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:54:59.085 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:54:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:54:59.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:59.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:54:59.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:54:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:54:59.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T17:54:59.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:54:59.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:54:59.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:54:59.799 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:54:59.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:54:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:54:59 smithi086 ceph-mon[29229]: pgmap v535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:59.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:54:59.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:54:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:54:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:54:59 smithi117 ceph-mon[32835]: pgmap v535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:54:59 smithi107 ceph-mon[32381]: pgmap v535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:54:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:54:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:54:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:54:59.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:00.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:00.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:00.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:00.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:00.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:00.761 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:55:00.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:00.792 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1604219585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:00.792 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:00 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:00.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1604219585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:00 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:01.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:01.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1604219585' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:01.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:00 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:01.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:01.249 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:01.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:01.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:01.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:01.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:01.486 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:01.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:01.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:01.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: osdmap e190: 12 total, 3 up, 11 in 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: osdmap e191: 12 total, 3 up, 11 in 2024-08-06T17:55:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:01 smithi117 ceph-mon[32835]: pgmap v538: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:01.917 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:01.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T17:55:01.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: osdmap e190: 12 total, 3 up, 11 in 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: osdmap e191: 12 total, 3 up, 11 in 2024-08-06T17:55:02.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:01 smithi086 ceph-mon[29229]: pgmap v538: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:02.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:02.203 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: osdmap e190: 12 total, 3 up, 11 in 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: osdmap e191: 12 total, 3 up, 11 in 2024-08-06T17:55:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:01 smithi107 ceph-mon[32381]: pgmap v538: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:02.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:02.341 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:02.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:02.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:02.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:02.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:02.342 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":191,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:01.219808+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:53:47.022044+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:02.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:02.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:02.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:02.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:02.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:03.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:03.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:03.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:03.337 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3732079252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:03.337 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:03.337 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:03.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:03.467 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3732079252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:03.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:55:03.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3732079252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:03.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:03.692 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:03.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:03.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:03.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:03.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:03.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:04.059 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:04.059 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:04.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:04.167 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:04.167 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:04.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:04.174 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:04.174 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:04.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:04.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:04.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:04.192 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:04.193 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:04.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:04.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:04.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:04.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:04.194 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:04.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:04.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T17:55:04.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:04 smithi086 ceph-mon[29229]: pgmap v539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:04.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3584678292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:04.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:04.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:04 smithi107 ceph-mon[32381]: pgmap v539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:04.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3584678292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:04.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:04 smithi117 ceph-mon[32835]: pgmap v539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3584678292' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:04.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:04.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:04.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:04.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:04.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:04.916 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:04.917 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:04.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:04.918 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:04.932 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:05.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:05.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:05.184 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:05.184 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:05.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:05.319 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.319 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.319 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:05.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:05.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:05.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:05.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:05.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:06.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:06.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:06.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:06.238 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:06.238 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:06.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:06 smithi086 ceph-mon[29229]: pgmap v540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:06.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:06 smithi117 ceph-mon[32835]: pgmap v540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:06.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:06.618 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:06.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:06 smithi107 ceph-mon[32381]: pgmap v540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:06.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:06.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:06.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:06.893 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:06.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:06.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:55:06.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:06.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:07.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:07.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:07.239 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:07.242 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:07.243 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:55:07.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:07.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:07.469 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:07.505 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:07.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:07.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:07.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:07.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:07.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:08.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:08.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:08.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:08.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:08.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:08 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:08.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:08.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:08 smithi086 ceph-mon[29229]: osdmap e192: 12 total, 3 up, 11 in 2024-08-06T17:55:08.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:08 smithi086 ceph-mon[29229]: pgmap v542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:08.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:08 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:08.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:08.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:08 smithi117 ceph-mon[32835]: osdmap e192: 12 total, 3 up, 11 in 2024-08-06T17:55:08.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:08 smithi117 ceph-mon[32835]: pgmap v542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:08.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:08 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1886581713' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:08 smithi107 ceph-mon[32381]: osdmap e192: 12 total, 3 up, 11 in 2024-08-06T17:55:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:08 smithi107 ceph-mon[32381]: pgmap v542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:08.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:08.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:08.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:08.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:09.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:09.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:09.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:09.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:09.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:09.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:09.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:09.730 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:09.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:09.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:10.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:10.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:10.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:10.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:10.500 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:10.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:10.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:55:10.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:55:10.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:55:10.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:55:10.590 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: osdmap e193: 12 total, 3 up, 11 in 2024-08-06T17:55:10.591 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:10 smithi117 ceph-mon[32835]: pgmap v544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: osdmap e193: 12 total, 3 up, 11 in 2024-08-06T17:55:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:10 smithi086 ceph-mon[29229]: pgmap v544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:10.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: osdmap e193: 12 total, 3 up, 11 in 2024-08-06T17:55:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:10 smithi107 ceph-mon[32381]: pgmap v544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:10.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:10.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:11.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:55:11.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:11.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:11.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:11.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:11.537 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:11.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T17:55:11.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:11.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:11.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:12.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:12.059 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:12.060 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:12.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:12.192 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:12.192 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:12.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:12.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:12 smithi107 ceph-mon[32381]: pgmap v545: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:12.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:12.453 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:12 smithi086 ceph-mon[29229]: pgmap v545: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:12.453 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:12.453 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:12.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:12.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:12 smithi117 ceph-mon[32835]: pgmap v545: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:12.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:12.844 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:12.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:13.091 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:13.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:13.248 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:13.248 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":193,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:09.454646+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:13.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:13.298 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:13.299 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:13.299 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:13.299 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:13.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:13.516 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3339836177' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:13.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:13.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:13.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:13.550 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3339836177' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3339836177' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:13.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:13.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:13.833 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:13.834 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:13.850 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:13.851 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:13.868 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:13.869 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:13.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:13.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:13.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:13.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:13.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:13.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:14.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:14.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:14.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:14.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:14.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:14.618 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:14.618 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:14.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:14 smithi086 ceph-mon[29229]: pgmap v546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:14.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/115183621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:14.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.619 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:14 smithi117 ceph-mon[32835]: pgmap v546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/115183621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:14 smithi107 ceph-mon[32381]: pgmap v546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/115183621' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.624 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:14.702 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:14.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:14.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:15.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:15.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:15.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:15.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:15.417 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:15.417 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:15.417 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:15.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:15.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:15.680 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:15.680 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:15.681 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:15.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:15.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:16.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:16.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:16.309 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:16.309 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:16.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:16.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:16.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:16.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:16.482 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:16 smithi107 ceph-mon[32381]: pgmap v547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:16.482 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:16.482 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:16.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:16.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:16 smithi086 ceph-mon[29229]: pgmap v547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:16.617 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:16 smithi117 ceph-mon[32835]: pgmap v547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:16.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:16.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:16.978 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:16.979 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:16.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:17.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:17.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:17.258 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:17.258 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:17.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:17.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:17.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:17.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:17 smithi117 ceph-mon[32835]: pgmap v548: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:17.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:17.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:17 smithi086 ceph-mon[29229]: pgmap v548: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:18.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:18.089 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:17 smithi107 ceph-mon[32381]: pgmap v548: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:18.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:18.296 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:18.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:18.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:55:18.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:18.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:18.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:18.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:18.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:18.761 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:18.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:18.928 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:18.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T17:55:19.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:19.034 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:19.137 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:55:19.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:19.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:19.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:19.428 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:19.428 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:19.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:19.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:19.706 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:19.706 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:19.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:19 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:55:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:19 smithi117 ceph-mon[32835]: osdmap e194: 12 total, 3 up, 11 in 2024-08-06T17:55:19.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:19 smithi117 ceph-mon[32835]: pgmap v550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:19.920 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:19.932 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:19.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:19.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:19 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:55:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:19 smithi107 ceph-mon[32381]: osdmap e194: 12 total, 3 up, 11 in 2024-08-06T17:55:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:19 smithi107 ceph-mon[32381]: pgmap v550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:20.047 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:20.047 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:20.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:19 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:55:20.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3803460554' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:20.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:19 smithi086 ceph-mon[29229]: osdmap e194: 12 total, 3 up, 11 in 2024-08-06T17:55:20.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:19 smithi086 ceph-mon[29229]: pgmap v550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:20.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:20.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:20.307 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:20.307 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:20.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:20.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:20.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:20.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:20.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:20.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:20.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:20.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:21.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:21.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:21.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:21.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:21.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:21.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:21.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:21.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:21.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:55:21.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:21.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:21.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:21.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:21.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:22.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:22.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:22.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:22 smithi107 ceph-mon[32381]: osdmap e195: 12 total, 3 up, 11 in 2024-08-06T17:55:22.254 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:22 smithi107 ceph-mon[32381]: pgmap v552: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:22.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:22.418 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:22.418 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:22.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:22 smithi086 ceph-mon[29229]: osdmap e195: 12 total, 3 up, 11 in 2024-08-06T17:55:22.418 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:22 smithi086 ceph-mon[29229]: pgmap v552: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:22.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:22.517 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:22 smithi117 ceph-mon[32835]: osdmap e195: 12 total, 3 up, 11 in 2024-08-06T17:55:22.517 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:22 smithi117 ceph-mon[32835]: pgmap v552: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:22.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:22.652 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:22.652 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:22.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:22.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:22.963 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:22.964 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":195,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:21.224372+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:54:17.592353+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:23.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:23.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:23.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:23.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:23.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:23.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:23.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/522673941' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:23.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:23.582 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/522673941' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:23.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/522673941' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:23.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:23.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:23.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:24.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:24.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:24.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:24.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:24 smithi086 ceph-mon[29229]: pgmap v553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:24.494 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:24.521 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:24 smithi117 ceph-mon[32835]: pgmap v553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:24.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:24.657 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:24 smithi107 ceph-mon[32381]: pgmap v553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:24.719 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:24.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:24.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:24.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:24.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:25.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:25.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:25.223 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:25.224 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:25.224 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:25.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:25.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:25.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:25.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:25.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:25.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:25.598 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:25.598 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:25.598 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:25.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:25.616 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:25.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:25.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:25.635 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:25.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:25.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:25.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:25.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:25.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:25.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:25.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3161094918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:25.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:25.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3161094918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:25.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:25.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3161094918' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:26.056 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:26.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T17:55:26.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:26.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:26.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:26.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:26.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:26 smithi107 ceph-mon[32381]: pgmap v554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:26.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:26.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:26.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:26 smithi086 ceph-mon[29229]: pgmap v554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:26.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:26.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:26.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:26 smithi117 ceph-mon[32835]: pgmap v554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:26.978 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:26.978 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:27.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:27.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:27.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:27.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:27.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:27.454 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:27.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:27.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:27.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:27 smithi086 ceph-mon[29229]: pgmap v555: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:27 smithi117 ceph-mon[32835]: pgmap v555: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:27.920 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:27.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:28.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:28.149 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:27 smithi107 ceph-mon[32381]: pgmap v555: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:28.149 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:28.150 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:28.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:28.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:28.419 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:28.419 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:28.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:28.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:28.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:28.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:28.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:55:29.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:29.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:29.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:29.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:29.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:29.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:29.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:30.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:30.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:29 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:55:30.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:29 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:55:30.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:29 smithi086 ceph-mon[29229]: osdmap e196: 12 total, 3 up, 11 in 2024-08-06T17:55:30.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:29 smithi086 ceph-mon[29229]: pgmap v557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:29 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:55:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:29 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:55:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:29 smithi117 ceph-mon[32835]: osdmap e196: 12 total, 3 up, 11 in 2024-08-06T17:55:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:29 smithi117 ceph-mon[32835]: pgmap v557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:30.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:30.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:30.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:29 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:55:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:29 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:55:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:29 smithi107 ceph-mon[32381]: osdmap e196: 12 total, 3 up, 11 in 2024-08-06T17:55:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:29 smithi107 ceph-mon[32381]: pgmap v557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:30.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:30.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:30.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:30.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:30.776 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:55:30.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:30.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:55:30.916 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:30.917 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":197,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:30.771835+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:30.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:31.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:55:31.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:31.101 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:55:31.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:31.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:31.370 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:31.370 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:31.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:31.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:31.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:31.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:31.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:31.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: osdmap e197: 12 total, 3 up, 11 in 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2418928217' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:31.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:31 smithi086 ceph-mon[29229]: pgmap v559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:32.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:32.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:32.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:55:32.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: osdmap e197: 12 total, 3 up, 11 in 2024-08-06T17:55:32.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2418928217' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:32.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:32.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:31 smithi107 ceph-mon[32381]: pgmap v559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3334734242' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:55:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: osdmap e197: 12 total, 3 up, 11 in 2024-08-06T17:55:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2418928217' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:31 smithi117 ceph-mon[32835]: pgmap v559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:32.127 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:32.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:32.233 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:32.233 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:32.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:32.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:32.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:32.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:32.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:55:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:55:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:32.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:33.030 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:33.030 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:33.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:33.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:33.130 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:55:33.130 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:55:33.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:55:33.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:55:33.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:33.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:33.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:33.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:33.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:33.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:33.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:33.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:33.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:33.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T17:55:33.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:33 smithi117 ceph-mon[32835]: pgmap v560: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:33.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:34.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:33 smithi086 ceph-mon[29229]: pgmap v560: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:34.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:34.131 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:33 smithi107 ceph-mon[32381]: pgmap v560: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:34.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:34.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:34.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:34.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:34.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:34.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:34.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:34.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:34.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:34.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:55:35.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:35.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:35.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:35.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:35.586 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:35.586 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:35.586 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:35.591 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:35.591 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:35.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:35.604 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:35.605 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:35.605 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:35.605 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:35.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:35.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:35.629 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:35.630 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:35.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:35.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:35.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:35.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:35.632 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:35.640 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:35.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:35.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4209201710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:35.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:35.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4209201710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4209201710' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:35.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:36.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:36.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:36.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:36.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:36.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:36.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:36.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:36.577 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:36.577 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:36.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:36.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:36.828 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:36.828 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:36.828 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.829 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.829 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.829 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:36 smithi107 ceph-mon[32381]: pgmap v561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:36 smithi086 ceph-mon[29229]: pgmap v561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:36 smithi117 ceph-mon[32835]: pgmap v561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:36.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:36.983 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:36.983 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:37.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:37.103 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:37.103 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:37.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:37.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:37.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:37.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:37.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:37.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:37.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:37.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:38.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:38.042 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:37 smithi107 ceph-mon[32381]: pgmap v562: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:38.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:37 smithi086 ceph-mon[29229]: pgmap v562: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:37 smithi117 ceph-mon[32835]: pgmap v562: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:38.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:38.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:38.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:38.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:38.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:38.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:38.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:38.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:38.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:55:39.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:39.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:39.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:39.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:39.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:39.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:39.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:39.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:39 smithi086 ceph-mon[29229]: pgmap v563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:39.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:39 smithi107 ceph-mon[32381]: pgmap v563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:40.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:39 smithi117 ceph-mon[32835]: pgmap v563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:40.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:40.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:40.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:40.515 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:40.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:40.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:40.768 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:55:40.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:40.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:40.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:40.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:40.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:40.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:41.038 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:41.038 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":198,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:40.764546+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:41.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:41.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:41.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:41.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:41.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:41.512 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:41.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:41.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: osdmap e198: 12 total, 3 up, 11 in 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2135544831' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: osdmap e199: 12 total, 3 up, 11 in 2024-08-06T17:55:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:41 smithi086 ceph-mon[29229]: pgmap v566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:41.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: osdmap e198: 12 total, 3 up, 11 in 2024-08-06T17:55:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2135544831' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: osdmap e199: 12 total, 3 up, 11 in 2024-08-06T17:55:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:41 smithi107 ceph-mon[32381]: pgmap v566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:42.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/54504934' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: osdmap e198: 12 total, 3 up, 11 in 2024-08-06T17:55:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2135544831' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: osdmap e199: 12 total, 3 up, 11 in 2024-08-06T17:55:42.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:41 smithi117 ceph-mon[32835]: pgmap v566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:42.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:42.202 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:42.202 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:42.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:42.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:42.524 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:42.524 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:42.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:42.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:42.719 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:55:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:55:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T17:55:42.720 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T17:55:42.720 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:55:42.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:42.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:43.056 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:43.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:43.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:43.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:43.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:43.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:43.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:43.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:43.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:55:43.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:43.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:43 smithi086 ceph-mon[29229]: pgmap v567: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:43.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T17:55:43.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:44.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:44.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:43 smithi107 ceph-mon[32381]: pgmap v567: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:44.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:44.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:43 smithi117 ceph-mon[32835]: pgmap v567: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:44.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:44.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:44.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:44.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:44.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:44.466 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:44.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:44.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:44.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:44.571 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:44.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:44.722 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:44.856 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:44.856 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:44.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:44.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:45.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:45.176 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:45.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:45.196 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:45.197 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:45.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:45.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:45.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:45.209 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:45.210 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:45.210 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:45.210 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:45.213 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:45.214 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:45.214 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:45.214 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:45.214 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:45.215 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:45.215 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:45.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:45.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:45.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/632814824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:45.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:45.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/632814824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:45.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:45.547 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/632814824' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:45.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:45.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:45.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:45.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:45.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:45.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:46.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:46.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:46.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:46 smithi107 ceph-mon[32381]: pgmap v568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:46.467 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:46.467 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:46.467 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.467 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.467 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.467 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:46 smithi086 ceph-mon[29229]: pgmap v568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:46.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:46.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:46 smithi117 ceph-mon[32835]: pgmap v568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:46.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:46.825 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:46.826 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:46.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:46.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:46.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:47.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:47.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:47.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:47.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:47.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:47.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:47.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:47.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:47.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:47.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:47.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:47.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:48.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:47 smithi086 ceph-mon[29229]: pgmap v569: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:48.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:47 smithi117 ceph-mon[32835]: pgmap v569: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:48.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:47 smithi107 ceph-mon[32381]: pgmap v569: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:48.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:48.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:48.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:48.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:48.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:48.770 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:55:48.789 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:55:48.789 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:55:48.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:55:48.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:49.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:49.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:49.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:49.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:55:49.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:49.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:49.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:49.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:49.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:49.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:49.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:49.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:49.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:49.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:49 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:49 smithi107 ceph-mon[32381]: osdmap e200: 12 total, 3 up, 11 in 2024-08-06T17:55:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:49 smithi107 ceph-mon[32381]: pgmap v571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:50.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:49 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:50.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:50.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:49 smithi086 ceph-mon[29229]: osdmap e200: 12 total, 3 up, 11 in 2024-08-06T17:55:50.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:49 smithi086 ceph-mon[29229]: pgmap v571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:50.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:49 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:55:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/254817308' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:55:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:49 smithi117 ceph-mon[32835]: osdmap e200: 12 total, 3 up, 11 in 2024-08-06T17:55:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:49 smithi117 ceph-mon[32835]: pgmap v571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:50.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:50.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:50.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:50.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:50.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:50.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:55:50.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:50.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:50.931 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:55:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:55:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:55:51.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:51 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:55:51.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:51 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:55:51.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:51.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:51.319 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:51 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:55:51.319 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:51 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:55:51.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:51.448 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:51 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:55:51.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:51 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:55:51.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:51.637 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:55:51.637 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":201,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:55:51.231509+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:55:51.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:51.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:55:51.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T17:55:51.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:51.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:52.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:52.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:52.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:52.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:52 smithi086 ceph-mon[29229]: osdmap e201: 12 total, 3 up, 11 in 2024-08-06T17:55:52.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1969051309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:52.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:52 smithi086 ceph-mon[29229]: pgmap v573: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:52 smithi117 ceph-mon[32835]: osdmap e201: 12 total, 3 up, 11 in 2024-08-06T17:55:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1969051309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:52 smithi117 ceph-mon[32835]: pgmap v573: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:52.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:52.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:52 smithi107 ceph-mon[32381]: osdmap e201: 12 total, 3 up, 11 in 2024-08-06T17:55:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1969051309' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:55:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:52 smithi107 ceph-mon[32381]: pgmap v573: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:52.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:52.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:53.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:53.118 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:53.118 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:53.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:53.173 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:53.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:53.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:53.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:53.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:53.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:53.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:53 smithi086 ceph-mon[29229]: pgmap v574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:53.858 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:53.858 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:53.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:54.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:54.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:53 smithi117 ceph-mon[32835]: pgmap v574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:54.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:53 smithi107 ceph-mon[32381]: pgmap v574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:54.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:54.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:54.390 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:54.390 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:54.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:54.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:54.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:54.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:54.898 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:55.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:55.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:55:55.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:55.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:55.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:55:55.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:55.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:55.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:55 smithi086 ceph-mon[29229]: pgmap v575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:55.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:55.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:56.015 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:55 smithi107 ceph-mon[32381]: pgmap v575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:56.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:56.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:56.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:55 smithi117 ceph-mon[32835]: pgmap v575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:56.202 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:55:56.203 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:55:56.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:55:56.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:56.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:56.421 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:55:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:55:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:55:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:55:56.422 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:55:56.439 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:55:56.440 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:55:56.440 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:55:56.440 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:55:56.440 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:55:56.440 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:55:56.441 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:55:56.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:56.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:55:56.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/325392494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:56.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:57.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/325392494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:57.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:57.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/325392494' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:55:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:57.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:57.271 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:57.272 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:57.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:55:57.418 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:55:57.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:57.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:57.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:57.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:57.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:57.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:57.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:57.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:57 smithi117 ceph-mon[32835]: pgmap v576: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:57.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:58.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:58.066 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:58.066 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:58.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:58.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:58.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:57 smithi086 ceph-mon[29229]: pgmap v576: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:58.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:58.200 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:58.200 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:55:58.200 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:57 smithi107 ceph-mon[32381]: pgmap v576: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:55:58.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:55:58.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:55:58.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:55:58.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:55:58.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T17:55:58.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:58.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:58.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:58.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:58.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:58.773 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:55:58.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:55:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:58.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:59.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:59.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:55:59.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:55:59.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:59.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:55:59.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:55:59.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:55:59.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:55:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:55:59.556 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:55:59.577 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:59.584 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:55:59.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:55:59.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:55:59.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:55:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:59 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:55:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:55:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:59 smithi107 ceph-mon[32381]: osdmap e202: 12 total, 3 up, 11 in 2024-08-06T17:55:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:55:59 smithi107 ceph-mon[32381]: pgmap v578: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:00.009 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:00.009 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:00.010 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:00.010 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:55:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:59 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:56:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:59 smithi086 ceph-mon[29229]: osdmap e202: 12 total, 3 up, 11 in 2024-08-06T17:56:00.011 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:55:59 smithi086 ceph-mon[29229]: pgmap v578: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:00.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:59 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:56:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2314998700' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:59 smithi117 ceph-mon[32835]: osdmap e202: 12 total, 3 up, 11 in 2024-08-06T17:56:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:55:59 smithi117 ceph-mon[32835]: pgmap v578: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:00.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:00.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:00.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:00.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:00.720 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:00.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:00.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:01.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:01.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:01.333 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:01.333 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:01.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:01.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:01.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:01.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:56:01.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:01.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:56:01.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:01.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:01.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:02.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:02.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:02.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:02.457 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:02.457 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:02.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:02 smithi086 ceph-mon[29229]: osdmap e203: 12 total, 3 up, 11 in 2024-08-06T17:56:02.457 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:02 smithi086 ceph-mon[29229]: pgmap v580: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:02.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:02 smithi107 ceph-mon[32381]: osdmap e203: 12 total, 3 up, 11 in 2024-08-06T17:56:02.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:02 smithi107 ceph-mon[32381]: pgmap v580: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:02.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:02.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:02 smithi117 ceph-mon[32835]: osdmap e203: 12 total, 3 up, 11 in 2024-08-06T17:56:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:02 smithi117 ceph-mon[32835]: pgmap v580: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:02.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:02.854 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:02.854 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":203,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:01.233994+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:00.999630+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:02.854 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:02.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:02.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:02.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:03.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:03.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:03.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:03.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:03.287 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:03.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:03.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:03.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2589364955' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:03.561 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2589364955' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:03.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:03.696 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2589364955' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:03.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:03.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:03.983 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:03.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T17:56:04.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:04.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:04.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:04.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:04.496 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:04 smithi117 ceph-mon[32835]: pgmap v581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:04.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:04 smithi086 ceph-mon[29229]: pgmap v581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:04.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:04.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:04.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:04.616 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:04 smithi107 ceph-mon[32381]: pgmap v581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:04.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:04.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:05.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:05.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:05.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:05.311 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:05.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:05.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:05.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:05.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:05.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:06.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:06.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:05 smithi086 ceph-mon[29229]: pgmap v582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:06.096 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:06.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:06.102 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:05 smithi117 ceph-mon[32835]: pgmap v582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:06.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:06.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:05 smithi107 ceph-mon[32381]: pgmap v582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:06.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:06.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:06.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:06.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:06.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:06.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:06.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:07.005 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:07.005 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:07.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:07.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:07.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:07.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:07.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:07.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:07.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:07.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:07.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:07 smithi117 ceph-mon[32835]: pgmap v583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:07.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:07 smithi107 ceph-mon[32381]: pgmap v583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:08.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:07 smithi086 ceph-mon[29229]: pgmap v583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:08.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:08.232 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:08.232 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:08.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:08.249 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:08.250 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:08.250 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:08.250 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:08.250 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:08.267 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:08.268 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:08.268 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:08.268 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:08.268 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:08.268 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:08.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:08.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:08.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:08.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:08.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:08.767 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:08.768 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:08.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:09.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:09.039 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:09.039 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:09.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2594249743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:09.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2594249743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:09.180 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2594249743' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:09.180 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:09.314 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:09.315 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:09.418 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:09.418 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:09.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:09.559 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:09.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:09.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:09.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:09.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:09 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:56:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:09 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:56:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:09 smithi086 ceph-mon[29229]: pgmap v584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:09.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:09.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:09.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:09 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:56:09.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:09 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:56:09.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:09 smithi117 ceph-mon[32835]: pgmap v584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:10.088 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:10.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:10.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:10.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:10.126 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:09 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:56:10.126 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:09 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:56:10.126 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:09 smithi107 ceph-mon[32381]: pgmap v584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:10.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:10.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:10.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:10.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:10.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:10.779 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:56:10.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:10.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:10 smithi107 ceph-mon[32381]: osdmap e204: 12 total, 3 up, 11 in 2024-08-06T17:56:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:11.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:11.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:10 smithi086 ceph-mon[29229]: osdmap e204: 12 total, 3 up, 11 in 2024-08-06T17:56:11.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:10 smithi117 ceph-mon[32835]: osdmap e204: 12 total, 3 up, 11 in 2024-08-06T17:56:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:11.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:11.193 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:11.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:11.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:56:11.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:11.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:11.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:11.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:11.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:11 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:11 smithi086 ceph-mon[29229]: osdmap e205: 12 total, 3 up, 11 in 2024-08-06T17:56:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:11 smithi086 ceph-mon[29229]: pgmap v587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:11 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:11 smithi117 ceph-mon[32835]: osdmap e205: 12 total, 3 up, 11 in 2024-08-06T17:56:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:11 smithi117 ceph-mon[32835]: pgmap v587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:12.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:12.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:11 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:12.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/588786708' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:12.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:11 smithi107 ceph-mon[32381]: osdmap e205: 12 total, 3 up, 11 in 2024-08-06T17:56:12.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:11 smithi107 ceph-mon[32381]: pgmap v587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:12.152 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:12.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T17:56:12.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:12.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:12.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:12.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:12.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:12.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:13.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:13.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:13.275 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:13.275 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:13.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:13.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:13.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:13.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:13.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:13.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:13.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:13.627 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:13.627 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:13.690 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:13.691 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":205,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:10.774873+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:09.403941+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:13.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:13.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:13.888 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1183836494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:13.888 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:13 smithi086 ceph-mon[29229]: pgmap v588: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:13.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1183836494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:13.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:13 smithi107 ceph-mon[32381]: pgmap v588: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:14.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:14.032 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1183836494' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:14.032 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:13 smithi117 ceph-mon[32835]: pgmap v588: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:14.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:14.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:14.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:14.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:14.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:56:14.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:14.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:14.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:15.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:15.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:15.284 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:15.285 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:15.285 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:15.285 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:15.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:15.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:15.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:15.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:15.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:15.996 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:16.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:16.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:15 smithi117 ceph-mon[32835]: pgmap v589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:16.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:15 smithi086 ceph-mon[29229]: pgmap v589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:16.098 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:16.098 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:16.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:16.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:15 smithi107 ceph-mon[32381]: pgmap v589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:16.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:16.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:16.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:16.676 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:16.677 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:16.693 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:16.694 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:16.695 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:16.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:16.711 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:16.712 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:16.712 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:16.712 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:16.713 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:16.713 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:16.713 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:16.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:16.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:17.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:17.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:17.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:17.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:17.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:17 smithi117 ceph-mon[32835]: from='client.15360 172.21.15.86:0/64898996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:17.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:17 smithi086 ceph-mon[29229]: from='client.15360 172.21.15.86:0/64898996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:17.668 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:17 smithi107 ceph-mon[32381]: from='client.15360 172.21.15.86:0/64898996' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:17.668 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.669 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:17.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:17.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:17.841 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:17.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:18.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:18.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:18.060 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:18.060 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:18.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:18.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:18.342 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:18.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:18 smithi086 ceph-mon[29229]: pgmap v590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:18 smithi117 ceph-mon[32835]: pgmap v590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:18.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:18.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:18.477 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:18 smithi107 ceph-mon[32381]: pgmap v590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:18.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:18.665 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:18.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:18.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:18.904 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:18.904 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:18.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:19.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:19.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:19.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:19.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:19.647 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:56:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:56:19.647 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T17:56:19.647 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:56:19.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:19.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:19.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:56:19.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:19 smithi086 ceph-mon[29229]: pgmap v591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:19.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:20.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:20.090 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:19 smithi107 ceph-mon[32381]: pgmap v591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:19 smithi117 ceph-mon[32835]: pgmap v591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:20.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:20.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:20.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:20.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:20.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:20.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:20.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:20.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:21.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:21.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:21.130 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":206,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:20.770009+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:21.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:21.238 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:56:21.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:21.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:21.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:21.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:21.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:56:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: osdmap e206: 12 total, 3 up, 11 in 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3938180511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: osdmap e207: 12 total, 3 up, 11 in 2024-08-06T17:56:21.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:21 smithi086 ceph-mon[29229]: pgmap v594: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:21.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: osdmap e206: 12 total, 3 up, 11 in 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3938180511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: osdmap e207: 12 total, 3 up, 11 in 2024-08-06T17:56:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:21 smithi107 ceph-mon[32381]: pgmap v594: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:22.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: osdmap e206: 12 total, 3 up, 11 in 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3938180511' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:22.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/107898895' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:22.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: osdmap e207: 12 total, 3 up, 11 in 2024-08-06T17:56:22.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:21 smithi117 ceph-mon[32835]: pgmap v594: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:22.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:22.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:22.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:22.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:22.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T17:56:22.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:22.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:22.688 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:22.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:22.782 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:22.782 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:22.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:22.917 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:22.917 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:23.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:23.113 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:23.114 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:23.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:23.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:23.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:23.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:23.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:23.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:23.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:23.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:23.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:24.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:56:24.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:23 smithi086 ceph-mon[29229]: pgmap v595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:24.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:23 smithi117 ceph-mon[32835]: pgmap v595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:23 smithi107 ceph-mon[32381]: pgmap v595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:24.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:24.337 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:24.338 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:24.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:24.405 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:24.415 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:24.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:24.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:24.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:24.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:25.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:25.098 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:25.098 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:25.098 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:25.101 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:25.102 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:25.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:25.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:25.103 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:25.120 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:25.122 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:25.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:25.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1758564574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:25.204 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:25.204 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:25.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1758564574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:25.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:25.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1758564574' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:25.472 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:25.472 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:25.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:25.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:25.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:25.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:26.000 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:26.000 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:26.000 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:26.000 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:26.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:26.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:26.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:26.254 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:26.254 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:26.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:26 smithi086 ceph-mon[29229]: pgmap v596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:26 smithi117 ceph-mon[32835]: pgmap v596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:26.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:26.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:26 smithi107 ceph-mon[32381]: pgmap v596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:26.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:26.521 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:26.521 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:26.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:26.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:26.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:26.925 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:26.926 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:26.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:26.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:27.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:27.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:27.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:27.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:27.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:27.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:27.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:27 smithi117 ceph-mon[32835]: pgmap v597: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:27.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:28.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:27 smithi086 ceph-mon[29229]: pgmap v597: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:28.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:27 smithi107 ceph-mon[32381]: pgmap v597: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:28.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:28.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:28.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:28.551 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:28.551 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:28.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:28.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:28.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:28.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:28.818 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:56:28.818 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:56:28.845 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:56:28.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:56:28.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:29.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:56:29.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:29.093 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:56:29.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:29.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:56:29.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:29.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:29.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:29.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:29.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:29.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:29.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T17:56:29.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:30.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:30.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:29 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:30.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:56:30.037 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:29 smithi117 ceph-mon[32835]: osdmap e208: 12 total, 3 up, 11 in 2024-08-06T17:56:30.037 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:29 smithi117 ceph-mon[32835]: pgmap v599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:30.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:30.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:30.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:29 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:30.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:56:30.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:29 smithi086 ceph-mon[29229]: osdmap e208: 12 total, 3 up, 11 in 2024-08-06T17:56:30.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:29 smithi086 ceph-mon[29229]: pgmap v599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:30.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:29 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2463179207' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:56:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:29 smithi107 ceph-mon[32381]: osdmap e208: 12 total, 3 up, 11 in 2024-08-06T17:56:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:29 smithi107 ceph-mon[32381]: pgmap v599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:30.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:30.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:30.294 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:30.299 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:30.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:30.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:30.549 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:30.549 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:30.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:30.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:30.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:31.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:31.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:31.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:31.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:31.545 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:31.546 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:31.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:31.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:31.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:32.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:32.155 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:32.155 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":209,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:31.241171+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:55:28.441940+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:32.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:32.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:32.320 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:32 smithi107 ceph-mon[32381]: osdmap e209: 12 total, 3 up, 11 in 2024-08-06T17:56:32.320 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:32 smithi107 ceph-mon[32381]: pgmap v601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:32.320 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2907916424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:32.321 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:32.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:56:32.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:32 smithi086 ceph-mon[29229]: osdmap e209: 12 total, 3 up, 11 in 2024-08-06T17:56:32.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:32 smithi086 ceph-mon[29229]: pgmap v601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:32.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2907916424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:32.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:32.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:32.563 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:32.585 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:32 smithi117 ceph-mon[32835]: osdmap e209: 12 total, 3 up, 11 in 2024-08-06T17:56:32.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:32 smithi117 ceph-mon[32835]: pgmap v601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:32.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2907916424' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:32.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:32.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:32.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:32.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:32.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:32.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:33.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:33.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:33.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:33.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:33.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:33.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:33.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:33.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:33.629 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:33.629 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:33.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:33.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:33 smithi086 ceph-mon[29229]: pgmap v602: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:33.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:33.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:33.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:33.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:33 smithi107 ceph-mon[32381]: pgmap v602: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:33.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:33 smithi117 ceph-mon[32835]: pgmap v602: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:34.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:34.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:34.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:34.414 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:34.414 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:34.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:34.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:34.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:34.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:35.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:35.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:35.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:35.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:35.591 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:35.591 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:35.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:35.668 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:35.669 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:35.669 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:35.685 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:35.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:35.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:35.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:35.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:35.706 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:35.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:35.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:35.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:35 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:56:35.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:35 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:56:35.741 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:35.741 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:35.741 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:35.741 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:35.741 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:35.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:35.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:35 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:56:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:35 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:56:35.883 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:35 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:56:35.883 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:35 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:56:35.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:36.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:36.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:36.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:36.462 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:36.462 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:36.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:36.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:36.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:36.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2509604806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:36 smithi107 ceph-mon[32381]: pgmap v603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:36.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:36.805 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:36.824 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:36.824 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:36.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2509604806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:36.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.824 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:36 smithi086 ceph-mon[29229]: pgmap v603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2509604806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:36 smithi117 ceph-mon[32835]: pgmap v603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:36.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:37.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:37.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:37.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:37.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:37.489 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:56:37.489 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T17:56:37.489 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T17:56:37.489 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T17:56:37.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:167,from:0,reservation_nonce:0)", 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 176.367482153, 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208364, 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:56:37.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7299999999999995e-07 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:56:37.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123197+0000", 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7089999999999998e-06 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123207+0000", 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123212+0000", 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6380000000000001e-06 2024-08-06T17:56:37.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123253+0000", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0308000000000001e-05 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123272+0000", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9191e-05 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123402+0000", 2024-08-06T17:56:37.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013016300000000001 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 176.36679415500001, 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8758999999999996e-05, 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:56:37.494 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7800000000000002e-07 2024-08-06T17:56:37.495 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123885+0000", 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9720000000000001e-06 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123892+0000", 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2119999999999997e-06 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.496 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123896+0000", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8580000000000002e-06 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123933+0000", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6906000000000002e-05 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123941+0000", 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8709999999999997e-06 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:56:37.497 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123960+0000", 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9462000000000001e-05 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:56:37.498 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:56:37.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:37.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:37.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:38.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:38.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:38.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:38.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:38.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T17:56:38.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:38.387 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:38.388 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:38.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:38.432 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:38.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:56:38.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:38.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:38.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:56:38.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:56:38.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:38 smithi086 ceph-mon[29229]: osdmap e210: 12 total, 3 up, 11 in 2024-08-06T17:56:38.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:38 smithi086 ceph-mon[29229]: pgmap v605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:38.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:38.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:56:38.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:56:38.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:38 smithi107 ceph-mon[32381]: osdmap e210: 12 total, 3 up, 11 in 2024-08-06T17:56:38.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:38 smithi107 ceph-mon[32381]: pgmap v605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:56:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:56:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:38 smithi117 ceph-mon[32835]: osdmap e210: 12 total, 3 up, 11 in 2024-08-06T17:56:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:38 smithi117 ceph-mon[32835]: pgmap v605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:38.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:39.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:39.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:39.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:39.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:39.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:39.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:39.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:39.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:39.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:39 smithi107 ceph-mon[32381]: pgmap v606: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:39.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:40.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:40.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:39 smithi086 ceph-mon[29229]: pgmap v606: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:40.116 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:39 smithi117 ceph-mon[32835]: pgmap v606: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:40.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:40.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:40.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:40.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:40.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:40.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:40.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:40.745 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:40.746 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:40.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T17:56:40.790 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:40.791 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:40.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:40.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:41.023 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:41.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:41.144 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:41.144 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:41.193 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:41.193 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":210,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:37.597338+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:41.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:41.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:41.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:41.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:41.660 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.660 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.660 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2082473414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:41.660 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:41 smithi117 ceph-mon[32835]: osdmap e211: 12 total, 3 up, 11 in 2024-08-06T17:56:41.660 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2082473414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:41 smithi107 ceph-mon[32381]: osdmap e211: 12 total, 3 up, 11 in 2024-08-06T17:56:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:41.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2082473414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:41 smithi086 ceph-mon[29229]: osdmap e211: 12 total, 3 up, 11 in 2024-08-06T17:56:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:41.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:42.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:42.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:42.245 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:56:42.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:42.377 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:42.377 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:42.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:42.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: pgmap v608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:42.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:42 smithi117 ceph-mon[32835]: osdmap e212: 12 total, 3 up, 11 in 2024-08-06T17:56:42.691 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:42.691 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: pgmap v608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:56:42.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:42.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:42 smithi107 ceph-mon[32381]: osdmap e212: 12 total, 3 up, 11 in 2024-08-06T17:56:42.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:42.782 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:42.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: pgmap v608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1187982943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:42 smithi086 ceph-mon[29229]: osdmap e212: 12 total, 3 up, 11 in 2024-08-06T17:56:42.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:43.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:43.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:43.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:43.407 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:43.407 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:43.514 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:43.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T17:56:43.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:43.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:43.762 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:43.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:43.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:43.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:43.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:43 smithi117 ceph-mon[32835]: pgmap v610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:43.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:44.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:43 smithi086 ceph-mon[29229]: pgmap v610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:44.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:44.089 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:43 smithi107 ceph-mon[32381]: pgmap v610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:44.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:44.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:44.424 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:44.424 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:44.424 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:44.440 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:44.441 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:44.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:44.458 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:44.458 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:44.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:44.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:44.738 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:44.738 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:44.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:44.872 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1432528900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:44.872 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:44.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:44.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:45.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1432528900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:45.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:45.100 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:45.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:45.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:45.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1432528900' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:45.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:45.124 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:45.124 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:45.178 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:45.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:45.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:45.448 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:45.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:45.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:45.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:45.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:56:45.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:46.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:46.083 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:45 smithi117 ceph-mon[32835]: pgmap v611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:46.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:45 smithi086 ceph-mon[29229]: pgmap v611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:46.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:46.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:45 smithi107 ceph-mon[32381]: pgmap v611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:46.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T17:56:46.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:46.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:46.424 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:46.470 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:46.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:46.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:46.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:46.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:46.951 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:46.951 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:47.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:47.101 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:47.101 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":213,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:46.245004+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:47.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:56:47.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:47.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:47.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:47.425 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:47 smithi107 ceph-mon[32381]: osdmap e213: 12 total, 3 up, 11 in 2024-08-06T17:56:47.425 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3944927735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:47.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:47.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:47.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:47 smithi086 ceph-mon[29229]: osdmap e213: 12 total, 3 up, 11 in 2024-08-06T17:56:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3944927735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:47 smithi117 ceph-mon[32835]: osdmap e213: 12 total, 3 up, 11 in 2024-08-06T17:56:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3944927735' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:47.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:47.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:47.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:47.797 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:47.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:47.948 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:47.948 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:48.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:48.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:48.204 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:48.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:48.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:48.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:48.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:48 smithi107 ceph-mon[32381]: pgmap v613: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:48.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:48.477 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:48.561 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:48 smithi086 ceph-mon[29229]: pgmap v613: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:48.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:48 smithi117 ceph-mon[32835]: pgmap v613: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:48.727 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:48.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:48.884 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:48.884 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:48.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:49.018 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:49.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T17:56:49.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:49.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:49.269 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:49.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:49.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:49.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:49.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:49.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:49.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:49 smithi117 ceph-mon[32835]: pgmap v614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:50.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:50.073 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:49 smithi107 ceph-mon[32381]: pgmap v614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:50.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:49 smithi086 ceph-mon[29229]: pgmap v614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:50.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:50.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:50.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:50.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:50.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:50.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:50.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:50.780 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:56:50.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:50.913 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:50.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:50.958 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:50.958 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:50.959 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:50.976 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:50.976 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:50.976 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:50.977 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:50.977 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:50.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:50.993 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:50.994 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:50.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:50.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:50.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:50.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:50.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:50.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:51.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:56:51.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:51.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:51.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:51.530 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:51.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:51.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:51.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T17:56:51.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:51.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:51.800 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:51.801 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: osdmap e214: 12 total, 3 up, 11 in 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1917704021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: osdmap e215: 12 total, 3 up, 11 in 2024-08-06T17:56:51.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:51 smithi086 ceph-mon[29229]: pgmap v617: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:51.840 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:51.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: osdmap e214: 12 total, 3 up, 11 in 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1917704021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: osdmap e215: 12 total, 3 up, 11 in 2024-08-06T17:56:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:51 smithi107 ceph-mon[32381]: pgmap v617: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:52.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1649885622' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:56:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: osdmap e214: 12 total, 3 up, 11 in 2024-08-06T17:56:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1917704021' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: osdmap e215: 12 total, 3 up, 11 in 2024-08-06T17:56:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:51 smithi117 ceph-mon[32835]: pgmap v617: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:52.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:52.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:52.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:52.469 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:52.470 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:52.526 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:52.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:52.589 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:52.589 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:52.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:52.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:52.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:52.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:53.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:53.208 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:53.209 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:53.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:53.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:53.441 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:56:53.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:53.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:53.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:53.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:53.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:53.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:56:53.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:53.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:53 smithi117 ceph-mon[32835]: pgmap v618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:54.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:53 smithi086 ceph-mon[29229]: pgmap v618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:54.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:54.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:53 smithi107 ceph-mon[32381]: pgmap v618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:54.114 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:56:54.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T17:56:54.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:54.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:54.334 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:54.362 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:54.434 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:56:54.434 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":215,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:56:51.246990+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:56:54.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:54.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:56:54.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:54.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:54.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/771148190' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:54.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:55.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/771148190' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:55.068 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/771148190' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:56:55.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:55.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:55.380 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:55.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:55.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:55.686 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:55.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:55.782 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:55.782 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:55.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:55.958 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:56:55.958 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T17:56:55.958 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T17:56:55.958 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:56:55.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:55 smithi107 ceph-mon[32381]: pgmap v619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:56.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:56.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:55 smithi086 ceph-mon[29229]: pgmap v619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:55 smithi117 ceph-mon[32835]: pgmap v619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:56.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:56.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:56.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:56:56.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:56.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T17:56:56.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:56.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:56.909 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:56.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:57.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:57.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:57.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:57.182 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:56:57.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:56:57.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:57.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:57.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:57.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:56:57.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:56:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:56:57.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:56:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:57.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:56:57.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:57.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:56:57.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:58.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:56:58.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:58.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:56:58.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:56:58.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:56:58.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:56:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:56:58.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:58 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:58.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:58.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:58 smithi086 ceph-mon[29229]: osdmap e216: 12 total, 3 up, 11 in 2024-08-06T17:56:58.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:58 smithi086 ceph-mon[29229]: pgmap v621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:58.424 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T17:56:58.424 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T17:56:58.424 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T17:56:58.424 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T17:56:58.424 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T17:56:58.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:58 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:58.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:58.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:58 smithi107 ceph-mon[32381]: osdmap e216: 12 total, 3 up, 11 in 2024-08-06T17:56:58.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:58 smithi107 ceph-mon[32381]: pgmap v621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:58.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:56:58.472 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:58 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:56:58.472 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3348919973' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:56:58.472 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:58 smithi117 ceph-mon[32835]: osdmap e216: 12 total, 3 up, 11 in 2024-08-06T17:56:58.472 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:58 smithi117 ceph-mon[32835]: pgmap v621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:56:58.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:58.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:58.832 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:56:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:56:58.832 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:56:58.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:58.849 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:56:58.849 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:56:58.850 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:56:58.850 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:56:58.850 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:56:58.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:56:58.867 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:56:58.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:56:58.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:56:58.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:56:58.868 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:56:58.868 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:56:58.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:59.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:56:59.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T17:56:59.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:59.256 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:59.257 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:59.257 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:56:59.257 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:56:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:56:59.257 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3394474268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:59.257 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.257 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:56:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:56:59.424 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:56:59.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:56:59.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3394474268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:59.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:56:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3394474268' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:56:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:56:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:56:59.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:56:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:56:59.670 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:56:59.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:56:59.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:56:59.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:00.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:00.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:00.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:00.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:00.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:00.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:00 smithi107 ceph-mon[32381]: pgmap v622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:00.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:00.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:00.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:00.586 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:00 smithi086 ceph-mon[29229]: pgmap v622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:00.587 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:00.587 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:00 smithi117 ceph-mon[32835]: pgmap v622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:00.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:00.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:00.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:57:00.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:01.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:01.122 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:57:01.122 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:57:01.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:01.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:01.480 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:01.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:01.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:01.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:01.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:01.647 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:01.647 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":217,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:01.249268+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:01.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:57:01.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:01.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:01.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:01.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T17:57:01.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:02.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:02.130 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:57:02.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:02.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:02.344 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:02.344 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:02.344 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:02 smithi107 ceph-mon[32381]: osdmap e217: 12 total, 3 up, 11 in 2024-08-06T17:57:02.344 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3239722281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:02.344 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:02 smithi107 ceph-mon[32381]: pgmap v624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:02.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:02.490 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:02 smithi117 ceph-mon[32835]: osdmap e217: 12 total, 3 up, 11 in 2024-08-06T17:57:02.491 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3239722281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:02.491 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:02 smithi117 ceph-mon[32835]: pgmap v624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:02.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:02 smithi086 ceph-mon[29229]: osdmap e217: 12 total, 3 up, 11 in 2024-08-06T17:57:02.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3239722281' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:02.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:02 smithi086 ceph-mon[29229]: pgmap v624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:02.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:02.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:02.757 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:02.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:02.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:03.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:03.192 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:03.192 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:03.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:03.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:03.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:03.539 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:03.539 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:03.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:03.706 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:03.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:03.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:03 smithi086 ceph-mon[29229]: pgmap v625: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:03.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:04.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:04.088 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:03 smithi107 ceph-mon[32381]: pgmap v625: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:03 smithi117 ceph-mon[32835]: pgmap v625: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:04.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:04.327 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:04.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T17:57:04.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:04.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:04.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:04.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:04.578 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:57:04.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:04.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:04.786 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:57:04.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:04.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:05.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:05.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:05.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:05.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:05.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:05.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:05.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:05.374 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:05.374 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:05.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:05.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:05.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:05.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:05.878 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:05 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:05.878 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:05.878 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:05 smithi086 ceph-mon[29229]: osdmap e218: 12 total, 3 up, 11 in 2024-08-06T17:57:05.878 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:05 smithi086 ceph-mon[29229]: pgmap v627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:05.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:06.011 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:06.011 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:06.011 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:06.012 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:06.012 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:06.012 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:06.012 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:06.012 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:06.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:05 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:06.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:06.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:05 smithi107 ceph-mon[32381]: osdmap e218: 12 total, 3 up, 11 in 2024-08-06T17:57:06.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:05 smithi107 ceph-mon[32381]: pgmap v627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:06.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:06.013 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:06.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:06.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:06.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:06.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:06.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:06.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:05 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1311297975' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:05 smithi117 ceph-mon[32835]: osdmap e218: 12 total, 3 up, 11 in 2024-08-06T17:57:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:05 smithi117 ceph-mon[32835]: pgmap v627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:06.228 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:06.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:06.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:06.399 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:06.399 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:06.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:06.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:06.744 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:06.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:06.814 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:06.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T17:57:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:06.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:06.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2731704681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:06.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:06.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:06.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:06.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:06 smithi086 ceph-mon[29229]: osdmap e219: 12 total, 3 up, 11 in 2024-08-06T17:57:07.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:07.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2731704681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:07.063 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.063 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.063 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.063 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:06 smithi107 ceph-mon[32381]: osdmap e219: 12 total, 3 up, 11 in 2024-08-06T17:57:07.107 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:57:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2731704681' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:06 smithi117 ceph-mon[32835]: osdmap e219: 12 total, 3 up, 11 in 2024-08-06T17:57:07.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:07.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:07.325 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:07.325 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:07.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:57:07.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:07.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:07.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:07.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:07 smithi086 ceph-mon[29229]: pgmap v629: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:07.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:07.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:07.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:08.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:07 smithi117 ceph-mon[32835]: pgmap v629: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:07 smithi107 ceph-mon[32381]: pgmap v629: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:08.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:08.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:08.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:08.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:08.701 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:08.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:57:08.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:08.838 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:08.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:08.938 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:08.939 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":219,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:06.251136+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:08.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:08.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:09.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:09.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:09.209 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2516644296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2516644296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:09.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:09.345 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2516644296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:09.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:09.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T17:57:09.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:09.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:09.650 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:09.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:09.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:09.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:09.993 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:09.994 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:10.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:09 smithi117 ceph-mon[32835]: pgmap v630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:10.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:09 smithi107 ceph-mon[32381]: pgmap v630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:10.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:10.272 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:09 smithi086 ceph-mon[29229]: pgmap v630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:10.374 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:10.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:10.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:10.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:10.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:10.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:10.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:10.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:11.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:11.179 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:11.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:11.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:11.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:11 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:57:11.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:11 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:57:11.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:11.332 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:11 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:57:11.333 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:11 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:57:11.333 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:11.333 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:11 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:57:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:11 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:57:11.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:11.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:11.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:11.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T17:57:11.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:11.963 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:11.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:12.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:12.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:12.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:12.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:12.519 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:12 smithi117 ceph-mon[32835]: pgmap v631: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:12.520 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:12.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:12 smithi086 ceph-mon[29229]: pgmap v631: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:12.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:12.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:12 smithi107 ceph-mon[32381]: pgmap v631: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:12.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:12.767 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:57:12.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:13.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:13.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:13.292 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:13 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:13.292 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:13.292 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:13 smithi117 ceph-mon[32835]: osdmap e220: 12 total, 3 up, 11 in 2024-08-06T17:57:13.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:13.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:13.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:13 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:13.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:13 smithi107 ceph-mon[32381]: osdmap e220: 12 total, 3 up, 11 in 2024-08-06T17:57:13.530 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:13.530 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:13.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:13 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:13.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3849933649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:13.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:13 smithi086 ceph-mon[29229]: osdmap e220: 12 total, 3 up, 11 in 2024-08-06T17:57:13.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:13.625 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:13.634 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:13.635 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:13.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:13.652 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:13.653 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:13.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:13.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:13.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:13.671 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:13.672 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:13.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:13.673 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:13.673 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:13.673 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:13.673 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:13.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:13.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:13.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:14.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:14.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:14.288 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:14.325 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:14.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:14.464 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:14.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T17:57:14.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:14.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/377592610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:14.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:14 smithi117 ceph-mon[32835]: pgmap v633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:14.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/377592610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:14.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:14 smithi086 ceph-mon[29229]: pgmap v633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:14.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:14.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:14.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/377592610' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:14 smithi107 ceph-mon[32381]: pgmap v633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:14.716 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:14.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:14.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:15.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:15.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:15.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:15.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:15.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:57:15.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:15.459 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:15.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:15.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:15.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:15.709 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:15.709 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:15.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:15.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:16.088 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:16.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:16.226 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:16.226 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":220,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:12.254248+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:16.239 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:16.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:16.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:57:16.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:16 smithi086 ceph-mon[29229]: pgmap v634: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:16.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1228318291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:16.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:16 smithi086 ceph-mon[29229]: osdmap e221: 12 total, 3 up, 11 in 2024-08-06T17:57:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:16 smithi117 ceph-mon[32835]: pgmap v634: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1228318291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:16 smithi117 ceph-mon[32835]: osdmap e221: 12 total, 3 up, 11 in 2024-08-06T17:57:16.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:16.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:16.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:16 smithi107 ceph-mon[32381]: pgmap v634: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:16.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1228318291' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:16.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:16 smithi107 ceph-mon[32381]: osdmap e221: 12 total, 3 up, 11 in 2024-08-06T17:57:16.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:16.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:16.807 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:16.807 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:16.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T17:57:16.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:17.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:17.115 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:17.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:17.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:17.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:17.538 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:17.538 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:17.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:17.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:17.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:17.937 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:17 smithi107 ceph-mon[32381]: pgmap v636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:17.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:18.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:17 smithi086 ceph-mon[29229]: pgmap v636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:18.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:17 smithi117 ceph-mon[32835]: pgmap v636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:18.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:18.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:18.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:18.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:18.732 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:18.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:18.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:18.942 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:57:18.947 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:18.947 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:18.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:19.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:19.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:19.210 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:19.210 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:19.247 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:19.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:19.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:19.463 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:19.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T17:57:19.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:19.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:19.761 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:19.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:19.862 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:19.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:19.863 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:19.863 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:19.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:19.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:20.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:19 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:19 smithi086 ceph-mon[29229]: osdmap e222: 12 total, 3 up, 11 in 2024-08-06T17:57:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:19 smithi086 ceph-mon[29229]: pgmap v638: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:20.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:19 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:20.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:20.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:19 smithi117 ceph-mon[32835]: osdmap e222: 12 total, 3 up, 11 in 2024-08-06T17:57:20.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:19 smithi117 ceph-mon[32835]: pgmap v638: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:20.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:19 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:20.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/911510969' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:20.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:19 smithi107 ceph-mon[32381]: osdmap e222: 12 total, 3 up, 11 in 2024-08-06T17:57:20.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:19 smithi107 ceph-mon[32381]: pgmap v638: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:20.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:20.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:20.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:20.599 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:20.599 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:20.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:20.697 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:20.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:20.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:20.715 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:20.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:20.715 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:20.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:20.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:20.723 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:20.734 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:20.734 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:20.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:20.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:20.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:20.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:20.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:20.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:20.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:21.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:21.044 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/949942042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:21.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:21.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/949942042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:21.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:21.353 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:21.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/949942042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:21.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:21.498 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:21.498 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:21.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:21.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:21.704 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:21.704 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:21.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:21.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:21.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:21.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:21.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:21.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:21.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:21 smithi086 ceph-mon[29229]: osdmap e223: 12 total, 3 up, 11 in 2024-08-06T17:57:21.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:22.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:22.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T17:57:22.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:21 smithi107 ceph-mon[32381]: osdmap e223: 12 total, 3 up, 11 in 2024-08-06T17:57:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:21 smithi107 ceph-mon[32381]: pgmap v640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:22.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:22.244 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.244 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.244 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:22.244 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:21 smithi117 ceph-mon[32835]: osdmap e223: 12 total, 3 up, 11 in 2024-08-06T17:57:22.244 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:21 smithi117 ceph-mon[32835]: pgmap v640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:22.294 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:22.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:57:22.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:22.361 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:21 smithi086 ceph-mon[29229]: pgmap v640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:22.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:22.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:22.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:22.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:23.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:23.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:23.272 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:23.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:23.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:23.440 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:23.440 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":223,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:21.255818+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:23.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:57:23.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:23.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/732840163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:23.569 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:23.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:23.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:23.697 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/732840163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:23.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:23.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/732840163' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:23.900 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:23.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:24.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:24.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:24.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:24.217 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:24.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:24.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:24.497 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:24.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T17:57:24.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:24.640 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:24 smithi107 ceph-mon[32381]: pgmap v641: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:24.741 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:24.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:24.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:24 smithi117 ceph-mon[32835]: pgmap v641: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:24.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:24 smithi086 ceph-mon[29229]: pgmap v641: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:24.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:24.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:24.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:25.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:25.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:25.248 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:25.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:25.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:25.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:25.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:25.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:25.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:25 smithi086 ceph-mon[29229]: pgmap v642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:25.825 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:25.825 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:25.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:26.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:25 smithi117 ceph-mon[32835]: pgmap v642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:26.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:26.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:25 smithi107 ceph-mon[32381]: pgmap v642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:26.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:26.339 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:26.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:26.377 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:26.377 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:26.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:26.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:26.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:26.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:26.968 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:26.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T17:57:26.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:26.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:27.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:27.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:27.218 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:27.238 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:57:27.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:27.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:27.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:27.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:27.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:27.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:27.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:27.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:27.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:27.670 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:27.670 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:27.671 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:27.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:27.687 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:27.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:27.706 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:27.706 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:27.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:27.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:27.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:28.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:28.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:28.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:28.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:28.547 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:28.547 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:28.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: osdmap e224: 12 total, 3 up, 11 in 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1978338624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.548 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:28 smithi086 ceph-mon[29229]: pgmap v644: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:28.561 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:28.561 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:28.562 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: osdmap e224: 12 total, 3 up, 11 in 2024-08-06T17:57:28.562 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1978338624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:28.562 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.562 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.562 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:28 smithi117 ceph-mon[32835]: pgmap v644: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:28.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2489883237' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: osdmap e224: 12 total, 3 up, 11 in 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1978338624' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:28.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:28 smithi107 ceph-mon[32381]: pgmap v644: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:28.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:28.861 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:28.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:28.926 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:28.935 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:28.935 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:28.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:29.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:29.142 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:29.143 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:29.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:29.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:29.411 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:29.411 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:29.411 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:29.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T17:57:29.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:29.520 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:29.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:29.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:29.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:29.744 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:29.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:29.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:30.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:30.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:30.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:30.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:30.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:30.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:30.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:30 smithi107 ceph-mon[32381]: pgmap v645: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:30.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:30.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:30.484 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:30 smithi086 ceph-mon[29229]: pgmap v645: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:30.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:57:30.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:30 smithi117 ceph-mon[32835]: pgmap v645: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:30.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:30.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:30.734 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:30.734 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:30.789 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:30.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:30.922 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:30.922 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":224,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:27.238493+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:08.906002+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:30.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:57:30.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:31.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:31.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:31.280 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:31.280 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:31.306 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:31.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:31.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/779927768' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:31.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:31 smithi117 ceph-mon[32835]: osdmap e225: 12 total, 3 up, 11 in 2024-08-06T17:57:31.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:31.530 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/779927768' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:31.530 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:31 smithi107 ceph-mon[32381]: osdmap e225: 12 total, 3 up, 11 in 2024-08-06T17:57:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/779927768' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:31 smithi086 ceph-mon[29229]: osdmap e225: 12 total, 3 up, 11 in 2024-08-06T17:57:31.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:31.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:31.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:31.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:31.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:31.973 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:31.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T17:57:32.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:32.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:32.224 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:32.230 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:32.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:32.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:32 smithi107 ceph-mon[32381]: pgmap v647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:32.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:32.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:32.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:32 smithi117 ceph-mon[32835]: pgmap v647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:32.570 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:32.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:32 smithi086 ceph-mon[29229]: pgmap v647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:32.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:32.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:32.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:33.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:33.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:33.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:33.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:33.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:33.763 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:33.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:33.784 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:33.784 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:33.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:34.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:34.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:57:34.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:34.334 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:34 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:57:34.334 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:34 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:57:34.334 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:34 smithi117 ceph-mon[32835]: osdmap e226: 12 total, 3 up, 11 in 2024-08-06T17:57:34.334 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:34 smithi117 ceph-mon[32835]: pgmap v649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:34.334 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:34.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:34.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:34 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:57:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:34 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:57:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:34 smithi086 ceph-mon[29229]: osdmap e226: 12 total, 3 up, 11 in 2024-08-06T17:57:34.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:34 smithi086 ceph-mon[29229]: pgmap v649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:34.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:34.410 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:34.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T17:57:34.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:34.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:34.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:34 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:57:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:34 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:57:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:34 smithi107 ceph-mon[32381]: osdmap e226: 12 total, 3 up, 11 in 2024-08-06T17:57:34.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:34 smithi107 ceph-mon[32381]: pgmap v649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:34.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:57:34.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:34.655 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:34.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:34.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:34.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:34.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:35.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:35.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:35.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:35 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:35 smithi117 ceph-mon[32835]: osdmap e227: 12 total, 3 up, 11 in 2024-08-06T17:57:35.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:35.526 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:35.526 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:35.526 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:35.545 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:35.545 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:35.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:35 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:35.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:35.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:35 smithi086 ceph-mon[29229]: osdmap e227: 12 total, 3 up, 11 in 2024-08-06T17:57:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:35.547 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:35.564 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:35.564 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:35.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:35.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:35.654 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:35 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:35.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4114923857' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:57:35.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:35 smithi107 ceph-mon[32381]: osdmap e227: 12 total, 3 up, 11 in 2024-08-06T17:57:35.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:35.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:36.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:36.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:36.202 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:36.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:36.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:36.277 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:36.278 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:36.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:36.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:36.506 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/465649582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:36.506 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.506 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.506 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.507 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:36 smithi086 ceph-mon[29229]: pgmap v651: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:36.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/465649582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:36.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:36 smithi117 ceph-mon[32835]: pgmap v651: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:36.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/465649582' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:36 smithi107 ceph-mon[32381]: pgmap v651: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:36.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:36.840 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:36.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T17:57:36.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:36.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:36.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:37.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:57:37.088 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:57:37.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:37.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:37.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:37.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:37.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:37.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:37.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:37.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:37.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:37.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:37.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:37.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:37 smithi117 ceph-mon[32835]: pgmap v652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:37.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:37 smithi086 ceph-mon[29229]: pgmap v652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:37.988 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:38.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:38.138 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:38.138 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":227,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:34.288589+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:37.590137+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:38.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:37 smithi107 ceph-mon[32381]: pgmap v652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:38.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:57:38.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:38.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:38.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:38.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:38.714 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:38.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2801631861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:38.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:38.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:38.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:39.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2801631861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:39.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2801631861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:39.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:39.253 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:39.253 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:39.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:39.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:39.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T17:57:39.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:39.518 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:39.519 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:39.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:39.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:39.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:39.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:39.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:39.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:39.918 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:39 smithi117 ceph-mon[32835]: pgmap v653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:40.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:40.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:39 smithi086 ceph-mon[29229]: pgmap v653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:40.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:40.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:39 smithi107 ceph-mon[32381]: pgmap v653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:40.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:40.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:40.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:40.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:40.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:40.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:41.015 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:41.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:41.177 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:57:41.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:41.241 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:41.243 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:41.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:41.372 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:57:41.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:41.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:41.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:41.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:41.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:42.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:42.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:42.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:42.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:42 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:42.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:42.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:42 smithi086 ceph-mon[29229]: osdmap e228: 12 total, 3 up, 11 in 2024-08-06T17:57:42.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:42 smithi086 ceph-mon[29229]: osdmap e229: 12 total, 3 up, 11 in 2024-08-06T17:57:42.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:42 smithi086 ceph-mon[29229]: pgmap v656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:42.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:42.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:42 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:42.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:42.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:42 smithi117 ceph-mon[32835]: osdmap e228: 12 total, 3 up, 11 in 2024-08-06T17:57:42.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:42 smithi117 ceph-mon[32835]: osdmap e229: 12 total, 3 up, 11 in 2024-08-06T17:57:42.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:42 smithi117 ceph-mon[32835]: pgmap v656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:42.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:42 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:57:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/951727510' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:57:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:42 smithi107 ceph-mon[32381]: osdmap e228: 12 total, 3 up, 11 in 2024-08-06T17:57:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:42 smithi107 ceph-mon[32381]: osdmap e229: 12 total, 3 up, 11 in 2024-08-06T17:57:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:42 smithi107 ceph-mon[32381]: pgmap v656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:42.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:42.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:42.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:42.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:43.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:43.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:43.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:43.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:43.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:43.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:57:43.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:57:43.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:43.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:57:43.535 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:43.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:43.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:43.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:43.920 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:43.920 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:44.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:44.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:44.163 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:44.163 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:44.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:44.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:44.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:44.667 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:44.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:44.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: pgmap v657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:57:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:57:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:44.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: pgmap v657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:44.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:57:44.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:57:44.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:44.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: pgmap v657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:57:44.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:57:44.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:57:44.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:45.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:57:45.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:45.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:45.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:45.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:45.346 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:45.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:45.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T17:57:45.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:45.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:45 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:57:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:45 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:57:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:45 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:57:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:45 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:57:45.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:45.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:45.883 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:45 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:57:45.883 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:45 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:57:45.948 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:45.949 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:45.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:45.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:45.981 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:45.982 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:45.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:45.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:45.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:45.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:45.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:45.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:46.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:46.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:46.277 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:46.278 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:46.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:46.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:46.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:46.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:46.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:46.823 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:46.823 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: osdmap e230: 12 total, 3 up, 11 in 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: pgmap v659: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2646511635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:46 smithi107 ceph-mon[32381]: osdmap e231: 12 total, 3 up, 11 in 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: osdmap e230: 12 total, 3 up, 11 in 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: pgmap v659: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2646511635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:46 smithi086 ceph-mon[29229]: osdmap e231: 12 total, 3 up, 11 in 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: osdmap e230: 12 total, 3 up, 11 in 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: pgmap v659: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2646511635' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:46.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:46 smithi117 ceph-mon[32835]: osdmap e231: 12 total, 3 up, 11 in 2024-08-06T17:57:46.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:47.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:47.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:47.209 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:47.209 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:47.302 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:47.303 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:47.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:47.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:47.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:47.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:47 smithi086 ceph-mon[29229]: pgmap v661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:47.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:47 smithi107 ceph-mon[32381]: pgmap v661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:48.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:47 smithi117 ceph-mon[32835]: pgmap v661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:48.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:48.222 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:48.223 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:48.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:48.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:48.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:48.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:48.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:48.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:57:48.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:48.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:48.982 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:48.982 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":231,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:46.261548+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:56:20.512024+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:49.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:49.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:49.313 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2608809657' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:49.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:49.366 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2608809657' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2608809657' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:49.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:49.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:49.639 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:49.639 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:49.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:49.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:50.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:50.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:50.150 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:50.167 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:50 smithi117 ceph-mon[32835]: pgmap v662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:50.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:50.310 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:50 smithi086 ceph-mon[29229]: pgmap v662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:50.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:50.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:50 smithi107 ceph-mon[32381]: pgmap v662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:50.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:50.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:50.573 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:50.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:50.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:50.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:51.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:51.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:51.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:51.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:51.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:51.380 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:51.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T17:57:51.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:51.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:51.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:51.790 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:51.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:52.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:52.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:52.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:52.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:52.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:52.287 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:52 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:57:52.287 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:52 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:57:52.287 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:52 smithi086 ceph-mon[29229]: osdmap e232: 12 total, 3 up, 11 in 2024-08-06T17:57:52.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:52 smithi086 ceph-mon[29229]: osdmap e233: 12 total, 3 up, 11 in 2024-08-06T17:57:52.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:52 smithi086 ceph-mon[29229]: pgmap v665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:52.311 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:52.311 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:52.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:52 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:57:52.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:52 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:57:52.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:52 smithi107 ceph-mon[32381]: osdmap e232: 12 total, 3 up, 11 in 2024-08-06T17:57:52.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:52 smithi107 ceph-mon[32381]: osdmap e233: 12 total, 3 up, 11 in 2024-08-06T17:57:52.312 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:52 smithi107 ceph-mon[32381]: pgmap v665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:52 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:57:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:52 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:57:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:52 smithi117 ceph-mon[32835]: osdmap e232: 12 total, 3 up, 11 in 2024-08-06T17:57:52.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:52 smithi117 ceph-mon[32835]: osdmap e233: 12 total, 3 up, 11 in 2024-08-06T17:57:52.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:52 smithi117 ceph-mon[32835]: pgmap v665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:52.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:52.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:52.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:52.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:57:52.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:52.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:53.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:53.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:53.281 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:57:53.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:53.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:53.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:53.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:53.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:57:53.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:53.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:54.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:54.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:54.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:54.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:54.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:54 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:54.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:54 smithi086 ceph-mon[29229]: osdmap e234: 12 total, 3 up, 11 in 2024-08-06T17:57:54.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:54 smithi086 ceph-mon[29229]: pgmap v667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:54.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:54.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:54.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:54 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:54 smithi107 ceph-mon[32381]: osdmap e234: 12 total, 3 up, 11 in 2024-08-06T17:57:54.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:54 smithi107 ceph-mon[32381]: pgmap v667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:54.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:54 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:57:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1514892694' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:57:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:54 smithi117 ceph-mon[32835]: osdmap e234: 12 total, 3 up, 11 in 2024-08-06T17:57:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:54 smithi117 ceph-mon[32835]: pgmap v667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:54.668 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:57:54.669 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:57:54.669 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:57:54.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:57:54.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:57:54.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:57:54.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:57:54.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:57:54.694 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:54.704 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:57:54.705 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:57:54.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:54.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:54.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:57:54.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:57:54.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:57:54.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:57:54.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:57:54.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:54.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:54.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:54.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:54.994 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:54.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:55.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:55.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:55.259 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:55.259 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:55.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:55.401 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2064264249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:55.401 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.401 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:55.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2064264249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:55.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.533 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:55.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:55.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:55.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2064264249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:57:55.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:55.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:55.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:56.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:56.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:57:56.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:56.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:56.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:57:56.336 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:57:56.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:57:56.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:56.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:56 smithi086 ceph-mon[29229]: pgmap v668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:56.514 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:56 smithi086 ceph-mon[29229]: osdmap e235: 12 total, 3 up, 11 in 2024-08-06T17:57:56.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:56.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:56.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:56 smithi117 ceph-mon[32835]: pgmap v668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:56.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:56 smithi117 ceph-mon[32835]: osdmap e235: 12 total, 3 up, 11 in 2024-08-06T17:57:56.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:57:56.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:56 smithi107 ceph-mon[32381]: pgmap v668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:56.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:56 smithi107 ceph-mon[32381]: osdmap e235: 12 total, 3 up, 11 in 2024-08-06T17:57:56.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:56.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:56.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:57.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:57.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:57.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:57.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:57:57.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:57.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:57.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:57 smithi086 ceph-mon[29229]: pgmap v670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:57.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:57:57.949 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:57 smithi117 ceph-mon[32835]: pgmap v670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:58.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:58.085 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:57 smithi107 ceph-mon[32381]: pgmap v670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:57:58.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:58.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:58.116 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:57:58.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:57:58.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:58.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:57:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:57:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:57:58.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:57:58.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:58.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:58.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:58.907 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:57:58.908 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":235,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:57:56.264752+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:50.746197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:57:58.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:57:59.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:59.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2671890629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:59.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2671890629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:59.248 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:57:59.248 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:57:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:57:59.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:57:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2671890629' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:57:59.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:57:59.502 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:57:59.502 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:57:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:57:59.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:57:59.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T17:57:59.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:57:59.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:57:59.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:57:59.915 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:57:59.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:57:59.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:00.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:00.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:57:59 smithi107 ceph-mon[32381]: pgmap v671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:00.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:00.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:57:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:00.191 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:57:59 smithi086 ceph-mon[29229]: pgmap v671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:00.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:00.232 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:57:59 smithi117 ceph-mon[32835]: pgmap v671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:00.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:00.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:00.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:00.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:00.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:00.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:00.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:00.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:00.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:01.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:01.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:01.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:01.301 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:01.301 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:01.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:01.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:01.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:01.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:01.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:02.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:02.070 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:02.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:02.260 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:02.260 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:02.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:02.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:02.488 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:02 smithi117 ceph-mon[32835]: pgmap v672: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:02.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:02.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:02 smithi086 ceph-mon[29229]: pgmap v672: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:02.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:02.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:02.622 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:02 smithi107 ceph-mon[32381]: pgmap v672: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:02.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:02.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:02.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:58:03.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:03.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:03.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:03.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:03.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:03.339 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:03.339 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:03.339 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:03.347 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:03.348 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:03.348 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:03.348 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:03.348 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:03.368 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:03.368 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:03.368 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:03.368 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:03.368 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:03.369 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:03.369 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:03.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:03.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:03.561 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:03 smithi117 ceph-mon[32835]: osdmap e236: 12 total, 3 up, 11 in 2024-08-06T17:58:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:03 smithi086 ceph-mon[29229]: osdmap e236: 12 total, 3 up, 11 in 2024-08-06T17:58:03.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:03 smithi107 ceph-mon[32381]: osdmap e236: 12 total, 3 up, 11 in 2024-08-06T17:58:03.805 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:03.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:04.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:04.094 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:04.094 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:04.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:04.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:04.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1615288489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:04.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:04 smithi086 ceph-mon[29229]: pgmap v674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:04.362 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1615288489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:04.362 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.362 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.362 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.362 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:04 smithi117 ceph-mon[32835]: pgmap v674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:04.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:04.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:04.632 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1615288489' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:04.632 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.632 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.632 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:04.633 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:04 smithi107 ceph-mon[32381]: pgmap v674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:04.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:04.765 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:04.765 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:58:04.773 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:58:04.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:05.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:05.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:05.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:05.279 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:05.280 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:05.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:05.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:05.286 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:58:05.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:05.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:05.557 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:05.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:05.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:05.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:05.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:05.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:05.739 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:05.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:05.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:05.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:06.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:06.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:06.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:06.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:06.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:06.350 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:06.351 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:06.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:06 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:06.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:06.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:06 smithi086 ceph-mon[29229]: osdmap e237: 12 total, 3 up, 11 in 2024-08-06T17:58:06.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:06 smithi086 ceph-mon[29229]: pgmap v676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:06 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:06 smithi117 ceph-mon[32835]: osdmap e237: 12 total, 3 up, 11 in 2024-08-06T17:58:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:06 smithi117 ceph-mon[32835]: pgmap v676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:06.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:06.569 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:06.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:06.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:06 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1131257114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:06 smithi107 ceph-mon[32381]: osdmap e237: 12 total, 3 up, 11 in 2024-08-06T17:58:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:06 smithi107 ceph-mon[32381]: pgmap v676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:06.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:06.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:06.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:07.011 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:07.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:07.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T17:58:07.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:07.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:07.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:07.523 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:07 smithi086 ceph-mon[29229]: osdmap e238: 12 total, 3 up, 11 in 2024-08-06T17:58:07.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:07.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:07 smithi117 ceph-mon[32835]: osdmap e238: 12 total, 3 up, 11 in 2024-08-06T17:58:07.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:07 smithi107 ceph-mon[32381]: osdmap e238: 12 total, 3 up, 11 in 2024-08-06T17:58:07.730 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:07.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:58:07.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:07.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:07.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:07.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:08.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:08.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:08.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:08.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:08 smithi117 ceph-mon[32835]: pgmap v678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:08.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:08.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:58:08.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:08 smithi086 ceph-mon[29229]: pgmap v678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:08.515 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:08.515 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:08.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:08 smithi107 ceph-mon[32381]: pgmap v678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:08.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:08.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:09.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:09.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:09.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:09.288 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:09.288 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:09.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:09.439 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:09.439 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:09.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:09.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:09.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:09.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:09 smithi086 ceph-mon[29229]: pgmap v679: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:09.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:09.834 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:09.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:10.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:10.100 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:09 smithi117 ceph-mon[32835]: pgmap v679: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:10.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:10.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:10.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:09 smithi107 ceph-mon[32381]: pgmap v679: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:10.288 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:10.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:10.475 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:58:10.476 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:58:10.478 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:58:10.478 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":238,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:58:06.288525+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:50.746197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:58:10.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:10.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:10.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:10.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2252638170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:11.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:11.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2252638170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2252638170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:11.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:11.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:11.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T17:58:11.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:11.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:11.683 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:11.683 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:11.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:11.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:11.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:12.014 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:12.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:12.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:12.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:12.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:12 smithi107 ceph-mon[32381]: osdmap e239: 12 total, 3 up, 11 in 2024-08-06T17:58:12.396 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:12 smithi107 ceph-mon[32381]: pgmap v681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:12.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:12.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:12 smithi117 ceph-mon[32835]: osdmap e239: 12 total, 3 up, 11 in 2024-08-06T17:58:12.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:12 smithi117 ceph-mon[32835]: pgmap v681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:12.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:12 smithi086 ceph-mon[29229]: osdmap e239: 12 total, 3 up, 11 in 2024-08-06T17:58:12.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:12 smithi086 ceph-mon[29229]: pgmap v681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:12.572 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:12.572 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:12.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:12.771 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:12.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:13.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:13.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:13.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:13.328 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:13.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:13.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:13.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:13.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:13.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:13 smithi107 ceph-mon[32381]: pgmap v682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:13.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:13 smithi086 ceph-mon[29229]: pgmap v682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:14.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:13 smithi117 ceph-mon[32835]: pgmap v682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:14.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:14.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:14.410 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:14.411 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:14.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:14.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:14.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:14.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:14.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:58:15.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:15.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:15.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:15.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:15.531 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:15.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:15.549 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:15.550 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:15.550 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:15.550 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:15.550 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:15.567 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:15.568 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:15.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:15.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:15.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:15.570 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:15.570 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:15.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2954839393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:15.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:15.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:15.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2954839393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:15.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:15.899 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2954839393' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:15.899 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:16.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:16.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:16.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:16.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:16.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:16.689 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:16.689 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:16.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:16 smithi086 ceph-mon[29229]: pgmap v683: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:16.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:16.824 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:16 smithi107 ceph-mon[32381]: pgmap v683: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:16 smithi117 ceph-mon[32835]: pgmap v683: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:16.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:17.027 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:17.065 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:17.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:17.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:17.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:17.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:219,from:6,reservation_nonce:6)", 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:57:10.175063+0000", 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.067862481999995, 2024-08-06T17:58:17.242 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014934, 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175063+0000", 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.243 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:58:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175063+0000", 2024-08-06T17:58:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.244 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175064+0000", 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5149999999999999e-06 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175073+0000", 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3850000000000002e-06 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:58:17.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175087+0000", 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4697e-05 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175112+0000", 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4652e-05 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175171+0000", 2024-08-06T17:58:17.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8795000000000002e-05 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175185+0000", 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4457000000000001e-05 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175212+0000", 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6838999999999998e-05 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:219/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:57:10.175820+0000", 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.067105660999999, 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013119499999999999, 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:58:17.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175820+0000", 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175820+0000", 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175820+0000", 2024-08-06T17:58:17.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8999999999999996e-07 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175829+0000", 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4810000000000006e-06 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175840+0000", 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1342000000000001e-05 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175846+0000", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4600000000000002e-06 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175900+0000", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4212999999999997e-05 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175913+0000", 2024-08-06T17:58:17.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2948999999999999e-05 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.175951+0000", 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8059999999999998e-05 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:219,from:6,reservation_nonce:0)", 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:57:10.176610+0000", 2024-08-06T17:58:17.252 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 67.066314735999995, 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000163504, 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176610+0000", 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176610+0000", 2024-08-06T17:58:17.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176611+0000", 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1600000000000002e-07 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176613+0000", 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2919999999999998e-06 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.254 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176619+0000", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1109999999999996e-06 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176625+0000", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1020000000000002e-06 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176747+0000", 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012220800000000001 2024-08-06T17:58:17.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176765+0000", 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7649999999999999e-05 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:57:10.176774+0000", 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8249999999999994e-06 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:58:17.256 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:58:17.257 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:58:17.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:17.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:17.571 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:17.571 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:17.575 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:58:17.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:17.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:17.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:58:17.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:58:17.854 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:17.854 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:17.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:17.882 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:17.882 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:58:18.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:18.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:18.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:18.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:18.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:18.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:18.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:18.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:18.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T17:58:18.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:18.834 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:18 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:18.835 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:58:18.835 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:18 smithi117 ceph-mon[32835]: osdmap e240: 12 total, 3 up, 11 in 2024-08-06T17:58:18.835 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:18 smithi117 ceph-mon[32835]: pgmap v685: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:18.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:18.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:18.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:18 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:18.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:58:18.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:18 smithi107 ceph-mon[32381]: osdmap e240: 12 total, 3 up, 11 in 2024-08-06T17:58:18.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:18 smithi107 ceph-mon[32381]: pgmap v685: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:18 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2980109477' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:58:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:18 smithi086 ceph-mon[29229]: osdmap e240: 12 total, 3 up, 11 in 2024-08-06T17:58:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:18 smithi086 ceph-mon[29229]: pgmap v685: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:18.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:19.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:19.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:19.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:19.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:19.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:19.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:19.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:19.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:19.779 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:19 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:58:19.780 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:19 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:58:19.780 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:19 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:58:19.780 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:19 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:58:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:19 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:58:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:19 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:58:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:19 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:58:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:19 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:58:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:19 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:58:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:19 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:58:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:19 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:58:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:19 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:58:19.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:58:19.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:20.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:20.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:20.176 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:20.176 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:20.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:20.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:20.424 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:20.428 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:20.446 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:20.447 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:20.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:20.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:20.701 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:20 smithi086 ceph-mon[29229]: pgmap v686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:20.701 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:20.702 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:20.702 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:20.702 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:20.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:20.869 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:20 smithi107 ceph-mon[32381]: pgmap v686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:20 smithi117 ceph-mon[32835]: pgmap v686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:20.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:20.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:20.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:21.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:21.137 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:21.137 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:21.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:21.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:21.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:21.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:58:21.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:21.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:21.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:22.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:22.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:22.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:22 smithi117 ceph-mon[32835]: osdmap e241: 12 total, 3 up, 11 in 2024-08-06T17:58:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:22 smithi117 ceph-mon[32835]: pgmap v688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:22.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:22.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:22.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:22 smithi086 ceph-mon[29229]: osdmap e241: 12 total, 3 up, 11 in 2024-08-06T17:58:22.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:22 smithi086 ceph-mon[29229]: pgmap v688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:22.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:22 smithi107 ceph-mon[32381]: osdmap e241: 12 total, 3 up, 11 in 2024-08-06T17:58:22.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:22 smithi107 ceph-mon[32381]: pgmap v688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:22.727 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:22.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:22.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:23.036 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:58:23.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:23.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:23.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:23.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:23.662 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:58:23.663 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":241,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:58:21.271198+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:50.746197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:58:23.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:23.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:23.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3416117102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:23.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3416117102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:58:24.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T17:58:24.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:24.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3416117102' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:24.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:24.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:24.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:24.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:24.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:24.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:24.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:24.770 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:24 smithi117 ceph-mon[32835]: pgmap v689: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:24 smithi086 ceph-mon[29229]: pgmap v689: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:24.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:24.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:24 smithi107 ceph-mon[32381]: pgmap v689: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:25.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:25.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:25.212 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:25.218 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:25.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:25.323 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:25.323 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:25.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:25.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:25.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:25.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:25.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:25 smithi086 ceph-mon[29229]: pgmap v690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:25.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:26.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:25 smithi117 ceph-mon[32835]: pgmap v690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:26.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:25 smithi107 ceph-mon[32381]: pgmap v690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:26.125 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:26.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:26.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:26.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:26.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:26.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:26.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:26.540 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:26.540 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:26.540 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:26.557 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:26.557 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:26.557 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:26.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:26.575 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:26.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:26.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:26.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:26.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:26.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:26.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:27.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:27.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4170390050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4170390050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:27.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4170390050' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:27.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:27.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:27.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:27.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:27.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:27.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:27.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:27.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:27.596 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:27.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:58:27.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:27.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:27.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:27 smithi086 ceph-mon[29229]: pgmap v691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:27.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:27.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:27.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:28.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.004 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:27 smithi117 ceph-mon[32835]: pgmap v691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:28.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:28.136 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:28.136 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:28.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:27 smithi107 ceph-mon[32381]: pgmap v691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:28.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:28.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:28.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:28.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:28.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:28.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:28.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:28.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:28.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:28.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:29.050 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:29.050 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:29.051 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:58:29.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:29.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:29.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:29.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:29.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:29.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:29.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:29.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:29.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:29.692 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:29.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:58:29.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T17:58:29.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:29.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:29.997 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:29.998 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:29 smithi086 ceph-mon[29229]: pgmap v692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:30.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:29 smithi117 ceph-mon[32835]: pgmap v692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:30.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:58:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:29 smithi107 ceph-mon[32381]: pgmap v692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:30.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:30.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:30.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:30.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:30.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:30.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:30.664 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:30.665 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:30.665 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:30.665 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:30.696 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:30.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:30.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:31.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:31.116 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:58:31.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:31.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:31.341 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:31.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:31.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:31.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:31.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:31.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:31.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:31.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:32.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:32.157 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:32.158 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:32.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:32 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:58:32.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:32.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:32 smithi086 ceph-mon[29229]: osdmap e242: 12 total, 3 up, 11 in 2024-08-06T17:58:32.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:32 smithi086 ceph-mon[29229]: osdmap e243: 12 total, 3 up, 11 in 2024-08-06T17:58:32.158 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:32 smithi086 ceph-mon[29229]: pgmap v695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:32.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:32.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:32.426 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:32 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:58:32.427 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:32.427 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:32 smithi117 ceph-mon[32835]: osdmap e242: 12 total, 3 up, 11 in 2024-08-06T17:58:32.427 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:32 smithi117 ceph-mon[32835]: osdmap e243: 12 total, 3 up, 11 in 2024-08-06T17:58:32.427 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:32 smithi117 ceph-mon[32835]: pgmap v695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:32 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:58:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2718635276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:32 smithi107 ceph-mon[32381]: osdmap e242: 12 total, 3 up, 11 in 2024-08-06T17:58:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:32 smithi107 ceph-mon[32381]: osdmap e243: 12 total, 3 up, 11 in 2024-08-06T17:58:32.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:32 smithi107 ceph-mon[32381]: pgmap v695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:32.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:32.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:32.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:32.846 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:32.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:33.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:33.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:33.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:33.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:33.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:33.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:33.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:33.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:33.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:34.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:34.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:34.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:34.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:34.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:34.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:34.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:34.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:34 smithi107 ceph-mon[32381]: pgmap v696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:34 smithi086 ceph-mon[29229]: pgmap v696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:34.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:35.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:34 smithi117 ceph-mon[32835]: pgmap v696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:35.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:35.281 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:58:35.301 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":243,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:58:31.274293+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:50.746197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:33.275865+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:58:35.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:35.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:35.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:58:35.590 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:35.591 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:35.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:35.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:35.841 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1853198926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:35.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:36.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:36.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1853198926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:36.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:36.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:36.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:36.182 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1853198926' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:36.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:36.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:36.467 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:36.468 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:36.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:36.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:36.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:36.568 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:58:36.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:36.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:36.928 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:36.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:36.930 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:36.931 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:36.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:37.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:37.113 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:36 smithi107 ceph-mon[32381]: pgmap v697: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:37.113 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3041433184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:37.113 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:36 smithi086 ceph-mon[29229]: pgmap v697: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3041433184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:36 smithi117 ceph-mon[32835]: pgmap v697: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3041433184' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:37.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:37.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:37.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:37.591 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:37.591 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:37.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:37.623 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:37.623 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:37.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:37.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:37.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:38.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:38.065 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:58:38.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T17:58:38.091 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:38.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:38.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:38.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:38.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:38.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:38.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:38.322 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:38.323 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:38.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:38.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:38.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:38.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:38 smithi117 ceph-mon[32835]: pgmap v698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:38.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:39.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:38 smithi086 ceph-mon[29229]: pgmap v698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:39.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:39.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:39.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:38 smithi107 ceph-mon[32381]: pgmap v698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:39.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:58:39.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:39.355 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:39.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:39.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:39.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:58:39.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:39.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:39.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:40.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:39 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:58:40.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:39 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:58:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:39 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:58:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:39 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:58:40.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:40.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:39 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:58:40.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:39 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:58:40.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:40.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:40.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:40.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:40.834 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:58:40.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:40 smithi107 ceph-mon[32381]: pgmap v699: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:40 smithi107 ceph-mon[32381]: osdmap e244: 12 total, 3 up, 11 in 2024-08-06T17:58:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:58:41.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:41.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:40 smithi117 ceph-mon[32835]: pgmap v699: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:41.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:40 smithi117 ceph-mon[32835]: osdmap e244: 12 total, 3 up, 11 in 2024-08-06T17:58:41.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:58:41.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:40 smithi086 ceph-mon[29229]: pgmap v699: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:41.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:40 smithi086 ceph-mon[29229]: osdmap e244: 12 total, 3 up, 11 in 2024-08-06T17:58:41.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:58:41.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:41.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:41.305 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:41.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:41.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:41.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:41.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:41.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:41.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:41.804 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:41.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:58:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:41 smithi107 ceph-mon[32381]: osdmap e245: 12 total, 3 up, 11 in 2024-08-06T17:58:42.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:42.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:42.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:58:42.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:41 smithi117 ceph-mon[32835]: osdmap e245: 12 total, 3 up, 11 in 2024-08-06T17:58:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/464048952' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:58:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:41 smithi086 ceph-mon[29229]: osdmap e245: 12 total, 3 up, 11 in 2024-08-06T17:58:42.117 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:42.117 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:42.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:42.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:42.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:42.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:42.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:42.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:42.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:42.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:43.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:42 smithi086 ceph-mon[29229]: pgmap v702: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:43.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:42 smithi117 ceph-mon[32835]: pgmap v702: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:43.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:42 smithi107 ceph-mon[32381]: pgmap v702: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:43.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:43.307 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:43.308 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:43.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:43.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:43.636 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:58:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T17:58:43.636 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T17:58:43.636 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:58:43.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:43.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:43.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:58:43.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:44.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:44.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:44.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:44.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:44.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:44.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T17:58:44.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:44.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:44.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:44.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:44.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:44 smithi086 ceph-mon[29229]: pgmap v703: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:44.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:58:44.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:58:44.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:58:44.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:58:45.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:44 smithi117 ceph-mon[32835]: pgmap v703: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:58:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:58:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:58:45.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:58:45.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:45.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:44 smithi107 ceph-mon[32381]: pgmap v703: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:45.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:58:45.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:58:45.157 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:58:45.157 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:58:45.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:45.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:45.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:45.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:45.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:45.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:45.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:45.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2706240595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:45.860 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:45.860 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:45.888 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:45.889 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:45.889 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:45.889 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:45.889 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:45.906 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:45.907 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:45.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:45.908 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:45.908 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:45.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:45.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:45.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:46.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2706240595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:46.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:46.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2706240595' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:46.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:46.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:46.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:46.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:46.484 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:46.484 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:46.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:46.612 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:58:46.612 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":245,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:58:40.828459+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:50.746197+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:57:44.925552+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:58:46.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:46.746 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:46.747 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:46.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:46.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:47.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:46 smithi086 ceph-mon[29229]: pgmap v704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:47.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4105971880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:46 smithi117 ceph-mon[32835]: pgmap v704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4105971880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:47.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:46 smithi107 ceph-mon[32381]: pgmap v704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4105971880' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:47.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:47.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:47.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:47.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:47.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:47.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:47.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:47 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:58:47.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:47 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:58:47.950 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:47.950 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:47.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:47 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:58:47.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:47 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:58:48.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:47 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:58:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:47 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:58:48.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:48.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:48.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:48.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:48.697 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:48.697 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:48.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:48.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:48.895 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:48.896 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:48.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:48 smithi086 ceph-mon[29229]: pgmap v705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:48.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:48 smithi086 ceph-mon[29229]: osdmap e246: 12 total, 3 up, 11 in 2024-08-06T17:58:49.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:48 smithi117 ceph-mon[32835]: pgmap v705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:48 smithi117 ceph-mon[32835]: osdmap e246: 12 total, 3 up, 11 in 2024-08-06T17:58:49.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:49.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:49.163 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:49.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:48 smithi107 ceph-mon[32381]: pgmap v705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:49.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:48 smithi107 ceph-mon[32381]: osdmap e246: 12 total, 3 up, 11 in 2024-08-06T17:58:49.215 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:49.216 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:49.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:49.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:49.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:49.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:49.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:49.824 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:49.824 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:49.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:50.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:50.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:50.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:50.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:58:50.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:50.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:50.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:58:50.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:50.862 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:58:50.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:50.953 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T17:58:50.953 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:58:50.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:50 smithi086 ceph-mon[29229]: pgmap v707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:50.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:51.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:51.042 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:50 smithi107 ceph-mon[32381]: pgmap v707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:51.043 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:50 smithi117 ceph-mon[32835]: pgmap v707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:58:51.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:51.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:51.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:51.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:51.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:51.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:51.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:51.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:52.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:51 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:52.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:51 smithi086 ceph-mon[29229]: osdmap e247: 12 total, 3 up, 11 in 2024-08-06T17:58:52.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:51 smithi086 ceph-mon[29229]: pgmap v709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:52.119 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:51 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:52.119 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:52.119 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:51 smithi117 ceph-mon[32835]: osdmap e247: 12 total, 3 up, 11 in 2024-08-06T17:58:52.119 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:51 smithi117 ceph-mon[32835]: pgmap v709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:51 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:58:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4191111612' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:58:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:51 smithi107 ceph-mon[32381]: osdmap e247: 12 total, 3 up, 11 in 2024-08-06T17:58:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:51 smithi107 ceph-mon[32381]: pgmap v709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:52.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:52.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:52.392 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:52.393 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:52.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T17:58:52.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:52.525 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:52.531 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:52.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:52.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:52.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:53.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:53.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:53.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:53.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:53.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:53.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:53.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:53.592 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:53 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:58:53.593 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:53 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:58:53.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:58:53.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:53.617 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:53.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:53 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:58:53.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:53 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:58:53.617 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:53.617 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:53.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:53 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:58:53.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:53 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:58:53.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:53.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:53.858 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:53.858 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:53.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:54.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:54.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:54.321 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:54.321 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:54.321 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:54 smithi086 ceph-mon[29229]: osdmap e248: 12 total, 3 up, 11 in 2024-08-06T17:58:54.322 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:54 smithi086 ceph-mon[29229]: pgmap v711: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:54.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:54.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:54.568 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:54.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:54.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:54 smithi117 ceph-mon[32835]: osdmap e248: 12 total, 3 up, 11 in 2024-08-06T17:58:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:54 smithi117 ceph-mon[32835]: pgmap v711: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:54.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:54.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:54 smithi107 ceph-mon[32381]: osdmap e248: 12 total, 3 up, 11 in 2024-08-06T17:58:54.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:54 smithi107 ceph-mon[32381]: pgmap v711: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:54.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:54.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:55.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:55.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:55.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:55.335 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:55.335 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:55.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:55.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:55.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:55.782 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:55.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:55.783 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:58:55.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:55.938 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:58:55.939 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":248,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:58:53.284477+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:58:56.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:55 smithi086 ceph-mon[29229]: pgmap v712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:56.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:56.080 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:55 smithi107 ceph-mon[32381]: pgmap v712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:55 smithi117 ceph-mon[32835]: pgmap v712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:56.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:56.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:56.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:58:56.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:56.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:56.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:56.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:56.814 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:58:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:58:56.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:58:56.833 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:58:56.834 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:58:56.834 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:58:56.834 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:58:56.834 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:58:56.852 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:58:56.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:58:56.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:58:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2854420702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:56 smithi117 ceph-mon[32835]: osdmap e249: 12 total, 3 up, 11 in 2024-08-06T17:58:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1590550877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:56.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:57.065 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:57.065 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:57.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2854420702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:57.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:56 smithi086 ceph-mon[29229]: osdmap e249: 12 total, 3 up, 11 in 2024-08-06T17:58:57.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1590550877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:57.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:57.160 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2854420702' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:58:57.160 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:56 smithi107 ceph-mon[32381]: osdmap e249: 12 total, 3 up, 11 in 2024-08-06T17:58:57.160 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1590550877' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:58:57.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:58:57.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:57.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:57.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:58:57.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:57.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:57.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:58:57.821 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:57.821 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:57.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:58.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:58.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:57 smithi086 ceph-mon[29229]: pgmap v714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:57 smithi117 ceph-mon[32835]: pgmap v714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:58:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:57 smithi107 ceph-mon[32381]: pgmap v714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:58.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:58:58.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:58.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:58.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:58:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:58:58.658 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:58:58.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:58:58.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:58:58.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:58:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:58:58.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:59.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:59.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:59.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:58:59.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:59.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:58:59.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:58:59.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:58:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:58:59 smithi086 ceph-mon[29229]: pgmap v715: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:59.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:58:59.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:58:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:58:59.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:58:59.984 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:58:59 smithi107 ceph-mon[32381]: pgmap v715: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:58:59.985 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:58:59.985 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:58:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:00.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:00.119 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:58:59 smithi117 ceph-mon[32835]: pgmap v715: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:00.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:00.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:00.471 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:00.471 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:00.471 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T17:59:00.471 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:167,from:0,reservation_nonce:0)", 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 319.34907661199998, 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208364, 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.472 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T17:59:00.473 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7299999999999995e-07 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123197+0000", 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7089999999999998e-06 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123207+0000", 2024-08-06T17:59:00.474 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123212+0000", 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6380000000000001e-06 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123253+0000", 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0308000000000001e-05 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.475 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123272+0000", 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9191e-05 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123402+0000", 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013016300000000001 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.476 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 319.34838861399999, 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8758999999999996e-05, 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:59:00.477 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T17:59:00.478 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7800000000000002e-07 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123885+0000", 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9720000000000001e-06 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123892+0000", 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2119999999999997e-06 2024-08-06T17:59:00.479 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123896+0000", 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8580000000000002e-06 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123933+0000", 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6906000000000002e-05 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.480 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123941+0000", 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8709999999999997e-06 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123960+0000", 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9462000000000001e-05 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.481 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:240,from:0,reservation_nonce:0)", 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.736386+0000", 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.735884718999998, 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019324500000000001, 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:59:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2099999999999995e-07 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.483 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736390+0000", 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3009999999999998e-06 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736400+0000", 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.065e-05 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:59:00.484 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736408+0000", 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6809999999999998e-06 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736448+0000", 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0349999999999998e-05 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736461+0000", 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3015e-05 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.485 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736579+0000", 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117427 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.737098+0000", 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 41.735172249999998, 2024-08-06T17:59:00.486 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5890999999999996e-05, 2024-08-06T17:59:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T17:59:00.487 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T17:59:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T17:59:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T17:59:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T17:59:00.489 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.53e-07 2024-08-06T17:59:00.490 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737101+0000", 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8830000000000002e-06 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737109+0000", 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6899999999999992e-06 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T17:59:00.491 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737113+0000", 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9489999999999998e-06 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737147+0000", 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4094000000000003e-05 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737154+0000", 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1760000000000004e-06 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T17:59:00.492 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737174+0000", 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9746e-05 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T17:59:00.493 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T17:59:00.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:00.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:00.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:59:00.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:00.825 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:59:00.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:01.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:01.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:01.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:01.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:01.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:01.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:01.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T17:59:01.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:01.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:01.614 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:01.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:01.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:01 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:01 smithi086 ceph-mon[29229]: osdmap e250: 12 total, 3 up, 11 in 2024-08-06T17:59:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:01 smithi086 ceph-mon[29229]: osdmap e251: 12 total, 3 up, 11 in 2024-08-06T17:59:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:01 smithi086 ceph-mon[29229]: pgmap v718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:01.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:01.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:01 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:01.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:01.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:01 smithi107 ceph-mon[32381]: osdmap e250: 12 total, 3 up, 11 in 2024-08-06T17:59:01.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:01 smithi107 ceph-mon[32381]: osdmap e251: 12 total, 3 up, 11 in 2024-08-06T17:59:01.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:01 smithi107 ceph-mon[32381]: pgmap v718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:02.076 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:02.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:01 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2497573478' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:01 smithi117 ceph-mon[32835]: osdmap e250: 12 total, 3 up, 11 in 2024-08-06T17:59:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:01 smithi117 ceph-mon[32835]: osdmap e251: 12 total, 3 up, 11 in 2024-08-06T17:59:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:01 smithi117 ceph-mon[32835]: pgmap v718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:02.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:02.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:02.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:02.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:02.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:02.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:02.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:02.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:02.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:02.783 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:02.783 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:02.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:02.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:03.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:03.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:03.199 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:03.291 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:03.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:03.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:03.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:03.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:03.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:03.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:03.853 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:03.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T17:59:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:03 smithi117 ceph-mon[32835]: pgmap v719: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:03.933 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:03.933 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:03.933 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:03.934 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:03.934 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:03.934 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:03.934 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:03.934 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:03.944 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:03.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:03.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:04.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:59:04.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:03 smithi086 ceph-mon[29229]: pgmap v719: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:04.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:04.110 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:04.198 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:03 smithi107 ceph-mon[32381]: pgmap v719: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:04.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:04.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:04.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:04.614 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:04.614 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:04.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:04.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:04.766 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:04.766 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:04.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:04.897 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1067203081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:04.898 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:04.898 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:04.898 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:05.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:05.029 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:05.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1067203081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:05.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1067203081' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:05.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:05.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:05.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:05.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:05.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:05.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:05.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:05.671 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:05.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:05.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:05 smithi107 ceph-mon[32381]: pgmap v720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:06.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:06.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:05 smithi086 ceph-mon[29229]: pgmap v720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:05 smithi117 ceph-mon[32835]: pgmap v720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:06.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:06.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:06.350 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:06.350 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:06.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:06.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T17:59:06.427 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:06.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:06.548 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":251,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:01.281979+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:06.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:06.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:59:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:06.636 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:06.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:06.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:06.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:06 smithi107 ceph-mon[32381]: from='client.15693 172.21.15.86:0/451410070' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:06.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:07.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:06 smithi086 ceph-mon[29229]: from='client.15693 172.21.15.86:0/451410070' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:07.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:06 smithi117 ceph-mon[32835]: from='client.15693 172.21.15.86:0/451410070' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:07.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:07.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:07.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:07.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:07.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:07.906 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:07.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:08.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:07 smithi086 ceph-mon[29229]: pgmap v721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:08.028 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:08.028 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:08.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:08.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:07 smithi107 ceph-mon[32381]: pgmap v721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:07 smithi117 ceph-mon[32835]: pgmap v721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:08.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:08.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:08.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:08.293 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:08.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:08.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:08.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:08.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:08.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:08.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:08.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=717047799,config=507150784,health=572275256,kv=1306471490,logm=2256341240}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90177692}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040414617}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1669716096}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766698520}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868236704}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079553714}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2493696662}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261101645}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277319425}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770643903}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705095800}) 2024-08-06T17:59:08.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2105166090}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3230535426}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125501607}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3375403917}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999130831}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652835761}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073106922}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91555705}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361483704}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321457696}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601962908}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266813305}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580778580}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590541550}) 2024-08-06T17:59:08.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=700717580}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762711735}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719768632}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760773174}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206153469}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16} crc {logm=2834879559,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1164459244}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=3815373383,mon_config_key=1844054997}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3822853416}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=80,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=2} crc {mon_config_key=2483393981,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=990368422,osdmap=3752773544}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1143642003}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=443960135}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191362387}) 2024-08-06T17:59:08.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:08 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3370097159}) 2024-08-06T17:59:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:08.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T17:59:08.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:08.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=717047799,config=507150784,health=572275256,kv=1306471490,logm=2256341240}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90177692}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040414617}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1669716096}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766698520}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868236704}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079553714}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2493696662}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261101645}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277319425}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770643903}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705095800}) 2024-08-06T17:59:08.996 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2105166090}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3230535426}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125501607}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3375403917}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999130831}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652835761}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073106922}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91555705}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361483704}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321457696}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601962908}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266813305}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580778580}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590541550}) 2024-08-06T17:59:08.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=700717580}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762711735}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719768632}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760773174}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206153469}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16} crc {logm=2834879559,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1164459244}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=3815373383,mon_config_key=1844054997}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3822853416}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=80,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=2} crc {mon_config_key=2483393981,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=990368422,osdmap=3752773544}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1143642003}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=443960135}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191362387}) 2024-08-06T17:59:08.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:08 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3370097159}) 2024-08-06T17:59:09.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:09.188 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=717047799,config=507150784,health=572275256,kv=1306471490,logm=2256341240}) 2024-08-06T17:59:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=90177692}) 2024-08-06T17:59:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2040414617}) 2024-08-06T17:59:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1669716096}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3766698520}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2868236704}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079553714}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2493696662}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3261101645}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=277319425}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=770643903}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1705095800}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2105166090}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3230535426}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2125501607}) 2024-08-06T17:59:09.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3375403917}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=999130831}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3652835761}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4073106922}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=91555705}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2361483704}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2321457696}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601962908}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3266813305}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1580778580}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2590541550}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=700717580}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2762711735}) 2024-08-06T17:59:09.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=719768632}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=760773174}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3206153469}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=60,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=16} crc {logm=2834879559,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1164459244}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mgrstat=1,mon_config_key=99} crc {mgrstat=3815373383,mon_config_key=1844054997}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3822853416}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=80,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=2} crc {mon_config_key=2483393981,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=990368422,osdmap=3752773544}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1143642003}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=443960135}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4191362387}) 2024-08-06T17:59:09.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:08 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=40} crc {osdmap=3370097159}) 2024-08-06T17:59:09.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:09.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:09.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:09.588 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:09.588 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:09.589 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:09.606 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:09.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:09.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:09.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:09.607 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:09.624 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:09.625 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:09.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:09.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:09.844 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:09.844 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:09.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:10.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:10.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/463018815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:10.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:09 smithi086 ceph-mon[29229]: pgmap v722: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/463018815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:09 smithi117 ceph-mon[32835]: pgmap v722: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:10.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:10.178 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/463018815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:10.178 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:09 smithi107 ceph-mon[32381]: pgmap v722: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:10.178 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:10.178 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:10.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:10.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:10.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:10.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:10.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:10.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:10.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:10.781 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:10.781 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:10.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:10.871 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:10.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:11.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.036 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:11.036 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:11.036 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:11.036 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:11.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:11.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:11.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:11.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:11.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:11.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:11.378 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:11.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:11.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:11.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:11.554 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:11.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:11.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T17:59:11.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:11.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:11.844 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:11.846 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:11.846 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:11.851 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:59:11.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:12.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:11 smithi086 ceph-mon[29229]: pgmap v723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:11 smithi117 ceph-mon[32835]: pgmap v723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:12.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:12.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:12.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:11 smithi107 ceph-mon[32381]: pgmap v723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:12.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:12.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:12.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:12.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:12.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:12.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:12.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:12.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:13.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:13.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:12 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:13.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:13.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:12 smithi086 ceph-mon[29229]: osdmap e252: 12 total, 3 up, 11 in 2024-08-06T17:59:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:12 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:12 smithi117 ceph-mon[32835]: osdmap e252: 12 total, 3 up, 11 in 2024-08-06T17:59:13.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:59:13.187 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:12 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:13.187 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3840409007' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:13.187 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:12 smithi107 ceph-mon[32381]: osdmap e252: 12 total, 3 up, 11 in 2024-08-06T17:59:13.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:13.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:13.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:13.486 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:13.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:13.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:13.753 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:13.853 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:13.853 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:13.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:13.939 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:13.939 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":252,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:11.847851+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:14.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:13 smithi086 ceph-mon[29229]: pgmap v725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:14.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:13 smithi107 ceph-mon[32381]: pgmap v725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:13 smithi117 ceph-mon[32835]: pgmap v725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:14.147 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:14.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T17:59:14.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:14.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:14.396 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:14.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:14.437 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:14.437 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:14.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:14.641 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:14.641 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:14.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:14.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:59:14.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:14.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:14.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2984766421' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:15.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:15.100 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2984766421' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:15.100 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:15.100 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2984766421' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:15.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:15.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:15.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:15.505 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:15.505 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:15.598 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:15.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:15.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:15.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:15.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:15.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:16.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:16.076 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:16.076 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:16.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:16.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:15 smithi086 ceph-mon[29229]: pgmap v726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:16.077 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:16.083 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:16.083 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:16.101 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:16.101 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:16.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:16.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:16.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:16.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:16.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:16.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:16.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:15 smithi107 ceph-mon[32381]: pgmap v726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:16.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:15 smithi117 ceph-mon[32835]: pgmap v726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:16.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:16.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:16.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:16.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:16.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:16.682 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:16.682 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:16.740 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:16.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T17:59:16.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:16.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:16.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3866708109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:16 smithi107 ceph-mon[32381]: osdmap e253: 12 total, 3 up, 11 in 2024-08-06T17:59:16.990 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:17.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:17.075 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:17.075 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:17.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3866708109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:17.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:17.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:17.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:16 smithi086 ceph-mon[29229]: osdmap e253: 12 total, 3 up, 11 in 2024-08-06T17:59:17.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:17.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3866708109' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:16 smithi117 ceph-mon[32835]: osdmap e253: 12 total, 3 up, 11 in 2024-08-06T17:59:17.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:17.522 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:17.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:17.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:17.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:17.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:17.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:18.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:18.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:18.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:17 smithi086 ceph-mon[29229]: pgmap v728: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:18.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:18.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:18.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:17 smithi117 ceph-mon[32835]: pgmap v728: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:18.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:18.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:17 smithi107 ceph-mon[32381]: pgmap v728: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:18.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:18.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:18.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:18.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:18.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:18.526 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:18.550 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:59:18.550 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:59:18.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:18.691 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:18.691 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:18.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:18.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:18.936 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:59:18.947 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:19.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:19.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T17:59:19.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:19.204 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:19.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:19.404 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:19.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:19.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:59:19.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:19.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:19.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:19.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:20.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:19 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:19 smithi086 ceph-mon[29229]: osdmap e254: 12 total, 3 up, 11 in 2024-08-06T17:59:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:19 smithi086 ceph-mon[29229]: pgmap v730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:20.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:19 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:20.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:20.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:19 smithi107 ceph-mon[32381]: osdmap e254: 12 total, 3 up, 11 in 2024-08-06T17:59:20.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:19 smithi107 ceph-mon[32381]: pgmap v730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:20.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:20.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:20.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:19 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:20.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/340846283' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:20.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:19 smithi117 ceph-mon[32835]: osdmap e254: 12 total, 3 up, 11 in 2024-08-06T17:59:20.375 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:19 smithi117 ceph-mon[32835]: pgmap v730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:20.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:20.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:20.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:20.903 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:20.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:21.043 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:21.043 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":254,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:18.931289+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:21.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:21.071 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:59:21.071 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:59:21.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:21.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:21.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3505006153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:21.448 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3505006153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:21.449 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3505006153' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:21.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:21.711 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:21.712 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:21.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:21.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:21.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T17:59:21.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:21.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:59:21.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:22.095 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:22.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:22.136 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:22.136 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:22.226 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:22.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:22.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:22.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:22 smithi107 ceph-mon[32381]: osdmap e255: 12 total, 3 up, 11 in 2024-08-06T17:59:22.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:22 smithi107 ceph-mon[32381]: pgmap v732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:22.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:22.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:22 smithi086 ceph-mon[29229]: osdmap e255: 12 total, 3 up, 11 in 2024-08-06T17:59:22.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:22 smithi086 ceph-mon[29229]: pgmap v732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:22.617 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:22 smithi117 ceph-mon[32835]: osdmap e255: 12 total, 3 up, 11 in 2024-08-06T17:59:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:22 smithi117 ceph-mon[32835]: pgmap v732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:22.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:22.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:22.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:22.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:22.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:23.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:23.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:23.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:23.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:23.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:23.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:23.796 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T17:59:23.796 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T17:59:23.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:23 smithi117 ceph-mon[32835]: pgmap v733: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1690237830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:23.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:23.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:23 smithi086 ceph-mon[29229]: pgmap v733: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:23.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1690237830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:24.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:23 smithi107 ceph-mon[32381]: pgmap v733: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1690237830' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:24.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:24.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:24.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:24.249 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:24.249 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:24.249 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:24.249 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:24.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:24.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:24.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T17:59:24.588 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:24.589 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:24.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:24.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:24.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:24.780 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:24.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:25.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:25.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:25.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.126 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:25.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:25.186 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:25.187 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:25.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:25.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:25.453 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:25.454 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:25.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:25.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:25.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:25.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:25.831 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:59:25.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:25 smithi117 ceph-mon[32835]: pgmap v734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:25.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:25.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:25 smithi107 ceph-mon[32381]: pgmap v734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:25.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:26.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:25 smithi086 ceph-mon[29229]: pgmap v734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:26.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:26.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:26.477 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:26.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:26.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:26.662 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:26.662 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:26.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:26.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:26 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:26 smithi107 ceph-mon[32381]: osdmap e256: 12 total, 3 up, 11 in 2024-08-06T17:59:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:26 smithi107 ceph-mon[32381]: osdmap e257: 12 total, 3 up, 11 in 2024-08-06T17:59:27.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:27.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:59:27.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:26 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:27.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:27.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:26 smithi086 ceph-mon[29229]: osdmap e256: 12 total, 3 up, 11 in 2024-08-06T17:59:27.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:26 smithi086 ceph-mon[29229]: osdmap e257: 12 total, 3 up, 11 in 2024-08-06T17:59:27.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:27.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:27.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:27.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:26 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3458316890' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:26 smithi117 ceph-mon[32835]: osdmap e256: 12 total, 3 up, 11 in 2024-08-06T17:59:27.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:26 smithi117 ceph-mon[32835]: osdmap e257: 12 total, 3 up, 11 in 2024-08-06T17:59:27.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:27.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T17:59:27.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:27.426 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:27.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:27.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:27.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:27.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:27.858 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:27 smithi117 ceph-mon[32835]: pgmap v737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:27.879 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:27.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:27.989 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:27 smithi086 ceph-mon[29229]: pgmap v737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:28.068 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:28.068 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":257,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:26.289023+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:28.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:27 smithi107 ceph-mon[32381]: pgmap v737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:28.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:28.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:28.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:28.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:28.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:28.514 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:28.514 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:28.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:28.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:28.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:28.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:59:28.935 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2721696713' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:29.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:29.061 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2721696713' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:29.115 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2721696713' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:29.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:29.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:29.347 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:29.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:29.452 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:29.452 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:29.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:29.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:29.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:29.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:29.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T17:59:29.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:29.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:29.993 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:29 smithi107 ceph-mon[32381]: pgmap v738: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:30.031 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T17:59:30.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:29 smithi086 ceph-mon[29229]: pgmap v738: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:30.067 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:30.067 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:30.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:29 smithi117 ceph-mon[32835]: pgmap v738: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:30.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:30.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:30.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:30.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:30.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:30.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:30.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:30.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:31.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:31.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:31.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:31.414 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:31.414 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:31.414 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:31.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:31.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:31.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:31.429 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:31.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:31.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:31.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:31.430 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:31.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:31.646 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:31.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:31.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:31.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:31.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:31.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:32.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:32.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:32.240 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:32.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:32.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:32.363 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:32.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/887453222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:59:32.364 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:32 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:59:32.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/887453222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:32.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.434 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:59:32.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:59:32.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:59:32.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:32 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:59:32.466 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:32.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T17:59:32.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/887453222' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T17:59:32.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:32 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T17:59:32.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:32.708 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:32.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:32.856 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:32.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:33.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:33.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:33.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:33.309 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T17:59:33.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:33.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:33.563 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:33.564 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:33.564 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:33.602 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:33.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T17:59:33.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:33.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:34.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:34.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:34.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:59:34.315 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:34.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:34.360 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:34 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:34.361 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:34.361 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:34 smithi107 ceph-mon[32381]: osdmap e258: 12 total, 3 up, 11 in 2024-08-06T17:59:34.361 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:34 smithi107 ceph-mon[32381]: pgmap v741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:34.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:34.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:34.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:34.492 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:34 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:34.492 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:34.492 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:34 smithi086 ceph-mon[29229]: osdmap e258: 12 total, 3 up, 11 in 2024-08-06T17:59:34.492 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:34 smithi086 ceph-mon[29229]: pgmap v741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:34.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:34.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:34 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:34.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3275512955' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T17:59:34.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:34 smithi117 ceph-mon[32835]: osdmap e258: 12 total, 3 up, 11 in 2024-08-06T17:59:34.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:34 smithi117 ceph-mon[32835]: pgmap v741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:34.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:34.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:34.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T17:59:35.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:35.140 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:35.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:35.202 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:35.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:35.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:35.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:35.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:35.325 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:35.356 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:35.356 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":258,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:33.302715+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:35.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:35.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/940123589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:35.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:35.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:59:35.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:59:35.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/940123589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:35.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T17:59:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T17:59:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T17:59:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/940123589' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:35.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:36.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T17:59:36.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:36.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:36.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:36.364 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:36.364 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:36.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:36.593 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:36.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:36.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:36.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:37.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:37.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:37.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:37.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:37.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:37.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T17:59:37.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:37.405 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:37.416 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:37 smithi086 ceph-mon[29229]: pgmap v742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:37.417 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:37 smithi086 ceph-mon[29229]: osdmap e259: 12 total, 3 up, 11 in 2024-08-06T17:59:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:37 smithi107 ceph-mon[32381]: pgmap v742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:37 smithi107 ceph-mon[32381]: osdmap e259: 12 total, 3 up, 11 in 2024-08-06T17:59:37.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:37.464 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:37.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:37.529 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:37 smithi117 ceph-mon[32835]: pgmap v742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:37 smithi117 ceph-mon[32835]: osdmap e259: 12 total, 3 up, 11 in 2024-08-06T17:59:37.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:37.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:37.859 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:37.860 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:37.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:37.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:37.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:37.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:37.884 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:37.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:38.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:38.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:38.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:38.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:38.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:38.338 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3481973402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:38.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.339 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:38.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:38.603 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3481973402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:38.603 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.603 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3481973402' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:38.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:38.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:38.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:38.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:39.028 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:39.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:39.109 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:39.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:39.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:39.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:39.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:39.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:39.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:39 smithi117 ceph-mon[32835]: pgmap v744: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:39.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:39.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:39.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:39.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:39 smithi086 ceph-mon[29229]: pgmap v744: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:39.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:39.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:39.686 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:39 smithi107 ceph-mon[32381]: pgmap v744: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:39.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:39.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T17:59:39.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:39.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:39.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:39.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:40.008 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:40.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:40.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:40.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:40.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:40.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:40.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:40.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:40.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:41.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:59:41.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:41.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:41.306 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T17:59:41.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:41.453 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:41.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:41.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:41 smithi086 ceph-mon[29229]: pgmap v745: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:41.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:41 smithi117 ceph-mon[32835]: pgmap v745: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:41.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:41.667 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:41 smithi107 ceph-mon[32381]: pgmap v745: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:41.667 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:41.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:41.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:41.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:41.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:42.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:42.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:42.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T17:59:42.170 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:42.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:42.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:42.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:42 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T17:59:42.328 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:42 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T17:59:42.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:42 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:42.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:42.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:42 smithi086 ceph-mon[29229]: osdmap e260: 12 total, 3 up, 11 in 2024-08-06T17:59:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:42 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T17:59:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:42 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T17:59:42.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:42 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:42.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:42.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:42 smithi117 ceph-mon[32835]: osdmap e260: 12 total, 3 up, 11 in 2024-08-06T17:59:42.382 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:42.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:42.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:42.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:42.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:42 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T17:59:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:42 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T17:59:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:42 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T17:59:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2821944147' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:42.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:42 smithi107 ceph-mon[32381]: osdmap e260: 12 total, 3 up, 11 in 2024-08-06T17:59:42.818 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:42.818 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":260,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:41.302504+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:39.505815+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:46.987575+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:42.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:42.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:43.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:43.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:43.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:43 smithi117 ceph-mon[32835]: pgmap v747: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:43 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T17:59:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:43 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T17:59:43.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4284649226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:43.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:43.524 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:43 smithi107 ceph-mon[32381]: pgmap v747: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:43.524 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:43 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T17:59:43.525 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:43 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T17:59:43.525 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4284649226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:43.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:43.586 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:43.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:43 smithi086 ceph-mon[29229]: pgmap v747: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:43.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:43 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T17:59:43.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:43 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T17:59:43.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4284649226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:43.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:43.770 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:43.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:43.991 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:44.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:44.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T17:59:44.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:44.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:44.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:44.455 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:44.455 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:44.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:44.631 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:44.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T17:59:44.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:44.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:44.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:44.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:44.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:44.879 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:44.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:45.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:45.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:45.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:45.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:45.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:45.251 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:45.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:45.268 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:45.268 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:45.268 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:45.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:45.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:45.286 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:45.287 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:45.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:45.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:45.288 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:45.288 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:45.288 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:45.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:45.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:45 smithi086 ceph-mon[29229]: pgmap v748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:45.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/748404070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:45.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:45.515 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:45 smithi117 ceph-mon[32835]: pgmap v748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:45.515 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/748404070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:45.542 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:45.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:45 smithi107 ceph-mon[32381]: pgmap v748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/748404070' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:45.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:45.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:45.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:45.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:46.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:46.067 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:46.068 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:46.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:46.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:46.304 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:46.324 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:46.324 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:46 smithi117 ceph-mon[32835]: osdmap e261: 12 total, 3 up, 11 in 2024-08-06T17:59:46.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:46.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:46.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.595 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:46 smithi107 ceph-mon[32381]: osdmap e261: 12 total, 3 up, 11 in 2024-08-06T17:59:46.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:46.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:46.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:46 smithi086 ceph-mon[29229]: osdmap e261: 12 total, 3 up, 11 in 2024-08-06T17:59:46.617 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:46.617 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:46.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:46.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:46.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:47.027 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:47.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T17:59:47.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:47.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:47.272 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:47.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:47 smithi086 ceph-mon[29229]: pgmap v749: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:47.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:47.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:47.534 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:47 smithi107 ceph-mon[32381]: pgmap v749: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:47.535 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:47.535 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:47.577 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:47 smithi117 ceph-mon[32835]: pgmap v749: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:47.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:47.666 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:47.666 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:47.725 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:47.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:47.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:48.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:48.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T17:59:48.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:48.193 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:48.194 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:48.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:48.333 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T17:59:48.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:48.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:48.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:48.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:48.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:48.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T17:59:48.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:48.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:48.727 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:48.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:48.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:48.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:48.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:49.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:49.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:49.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:49 smithi086 ceph-mon[29229]: pgmap v751: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:49.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:49 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:49.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:49.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:49 smithi086 ceph-mon[29229]: osdmap e262: 12 total, 3 up, 11 in 2024-08-06T17:59:49.421 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:49 smithi117 ceph-mon[32835]: pgmap v751: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:49.421 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:49 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:49.421 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:49.421 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:49 smithi117 ceph-mon[32835]: osdmap e262: 12 total, 3 up, 11 in 2024-08-06T17:59:49.448 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:49.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T17:59:49.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:49.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:49.649 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:49.649 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:49.696 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:49 smithi107 ceph-mon[32381]: pgmap v751: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:49 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1834412846' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T17:59:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:49 smithi107 ceph-mon[32381]: osdmap e262: 12 total, 3 up, 11 in 2024-08-06T17:59:49.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:49.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:50.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:50.103 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:50.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:50.273 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T17:59:50.274+0000 7f1492d0a640 0 --2- 172.21.15.86:0/1802051881 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f148c107d20 0x7f148c19de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T17:59:50.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:50.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:50.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:50 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:59:50.437 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:50 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:59:50.465 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:50.466 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":263,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:50.330473+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:58:52.641369+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:50.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:50.551 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:50 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:59:50.552 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:50 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:59:50.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:50 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:59:50.586 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:50 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:59:50.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:50.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:50.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:50.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:51.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:51.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T17:59:51.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:51.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:51.358 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:51.358 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:51.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:51 smithi107 ceph-mon[32381]: pgmap v753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:51.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:51 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:59:51.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:51 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:59:51.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:51 smithi107 ceph-mon[32381]: osdmap e263: 12 total, 3 up, 11 in 2024-08-06T17:59:51.359 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1802051881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:51 smithi117 ceph-mon[32835]: pgmap v753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:51 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:59:51.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:51 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:59:51.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:51 smithi117 ceph-mon[32835]: osdmap e263: 12 total, 3 up, 11 in 2024-08-06T17:59:51.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1802051881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:51.455 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:51.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:51.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:51 smithi086 ceph-mon[29229]: pgmap v753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:51.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:51 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:59:51.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:51 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:59:51.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:51 smithi086 ceph-mon[29229]: osdmap e263: 12 total, 3 up, 11 in 2024-08-06T17:59:51.489 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1802051881' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:51.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:51.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:51.740 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:51.740 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:51.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:51.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T17:59:51.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T17:59:51.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:52.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T17:59:52.041 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:52.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:52.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:52.284 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:52.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:52.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:52.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:52.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:52.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:52.936 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:52.936 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:53.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:53.159 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:53.159 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:53.159 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:53.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:53.166 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:53.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:53.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:53.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:53.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:53.184 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:53.184 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:53.184 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:53.184 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:53.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:53.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:53.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:53.188 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:53.189 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:53.189 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:53.189 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:53.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:53.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:53 smithi117 ceph-mon[32835]: pgmap v755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:53.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2024621395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:53.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:53.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:53.602 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:53 smithi086 ceph-mon[29229]: pgmap v755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:53.602 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2024621395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:53.602 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:53.602 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:53.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:53.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:53 smithi107 ceph-mon[32381]: pgmap v755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:53.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2024621395' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:53.773 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:53.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:53.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:53.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:53.923 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T17:59:53.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T17:59:53.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:54.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:54.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:54.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.383 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:54.383 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:54.384 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.384 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.384 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.469 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:54.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T17:59:54.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:54.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T17:59:54.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T17:59:54.717 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:54.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:54.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:54.941 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:55.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:55.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:55.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:55.345 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T17:59:55.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:55.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:55 smithi117 ceph-mon[32835]: pgmap v756: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:55.483 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:55.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:55.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:55 smithi086 ceph-mon[29229]: pgmap v756: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:55.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:55 smithi107 ceph-mon[32381]: pgmap v756: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T17:59:55.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:55.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:55.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T17:59:55.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:56.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:56.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:56.294 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:56.356 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:56.357 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:56.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:56.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: osdmap e264: 12 total, 3 up, 11 in 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T17:59:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:56 smithi086 ceph-mon[29229]: osdmap e265: 12 total, 3 up, 11 in 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: osdmap e264: 12 total, 3 up, 11 in 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T17:59:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:56 smithi117 ceph-mon[32835]: osdmap e265: 12 total, 3 up, 11 in 2024-08-06T17:59:56.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2564625225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: osdmap e264: 12 total, 3 up, 11 in 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T17:59:56.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:56 smithi107 ceph-mon[32381]: osdmap e265: 12 total, 3 up, 11 in 2024-08-06T17:59:56.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:56.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:57.033 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T17:59:57.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:57.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T17:59:57.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:57.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:57.283 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:57.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:57.438 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:57 smithi086 ceph-mon[29229]: pgmap v758: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:57.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:57.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T17:59:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:57 smithi117 ceph-mon[32835]: pgmap v758: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:57 smithi107 ceph-mon[32381]: pgmap v758: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:57.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T17:59:57.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:57.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:57.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T17:59:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T17:59:57.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:58.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:58.188 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T17:59:58.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:58.189 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:58.189 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":265,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T17:59:56.296789+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T17:59:58.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T17:59:58.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T17:59:58.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4012641571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:58.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:58.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4012641571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:58.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:58.673 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4012641571' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T17:59:58.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T17:59:58.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T17:59:58.857 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T17:59:58.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T17:59:59.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:59.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T17:59:59.244 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T17:59:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T17:59:59.244 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T17:59:59.263 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T17:59:59.263 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T17:59:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T17:59:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T17:59:59.264 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T17:59:59.281 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T17:59:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T17:59:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T17:59:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T17:59:59.282 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T17:59:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T17:59:59.283 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T17:59:59.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:59.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T17:59:59.561 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T17:59:59.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T17:59:59.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T17:59:59.584 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:59 smithi117 ceph-mon[32835]: pgmap v760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:59.584 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 17:59:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1098498130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:59.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:59 smithi086 ceph-mon[29229]: pgmap v760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 17:59:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1098498130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:59.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T17:59:59.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 17:59:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T17:59:59.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T17:59:59.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:59 smithi107 ceph-mon[32381]: pgmap v760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T17:59:59.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 17:59:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1098498130' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T17:59:59.809 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T17:59:59.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T17:59:59.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:00.040 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:00.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:00.151 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:00.151 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 17:59:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:00.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:00.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:00.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:00.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:00.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:00:00.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 1.0 is stuck stale for 9m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.2 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck inactive for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.6 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.8 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.9 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.a is stuck stale for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.465 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.c is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.d is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.e is stuck inactive for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.f is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.10 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.13 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.15 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.466 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.18 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck undersized for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.467 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.a is stuck undersized for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.c is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.e is stuck undersized for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck undersized for 8m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.468 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:00:00.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 1.0 is stuck stale for 9m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.2 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck inactive for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.534 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.6 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.8 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.9 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.a is stuck stale for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.c is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.d is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.e is stuck inactive for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.f is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.10 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.535 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.13 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.15 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.18 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.536 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck undersized for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.a is stuck undersized for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.c is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.e is stuck undersized for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.538 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.538 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.538 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck undersized for 8m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.538 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:00.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:00.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:00.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:00.693 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:00:00.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 1.0 is stuck stale for 9m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.2 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck inactive for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.695 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.6 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.8 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.9 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.a is stuck stale for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.c is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.d is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.e is stuck inactive for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.f is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.10 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.696 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.13 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.15 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.18 is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck inactive for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1b is stuck inactive for 9m, current state unknown, last acting [] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck stale for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck inactive for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck undersized for 9m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.a is stuck undersized for 8m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.c is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.698 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.e is stuck undersized for 8m, current state undersized+peered, last acting [2] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck undersized for 9m, current state undersized+peered, last acting [6] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck undersized for 9m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck undersized for 8m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck undersized for 9m, current state undersized+peered, last acting [0] 2024-08-06T18:00:00.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:00.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:01.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:01.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:01.328 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:01.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:01.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:01.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:01.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:01.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:01 smithi117 ceph-mon[32835]: pgmap v761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:01.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:01 smithi086 ceph-mon[29229]: pgmap v761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:01 smithi107 ceph-mon[32381]: pgmap v761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:01.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:01.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:02.002 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:02.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:00:02.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:02.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:02.260 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:02.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:02.376 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:00:02.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:02.518 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:02.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:02.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:02.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:02.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:00:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:02.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:02.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:02.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:00:02.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:02.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:03.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:03.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:03.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:03.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:03.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:03.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:03.491 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:03 smithi107 ceph-mon[32381]: pgmap v762: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:03.491 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:03 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:03.491 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:03.491 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:03 smithi107 ceph-mon[32381]: osdmap e266: 12 total, 3 up, 11 in 2024-08-06T18:00:03.601 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:03 smithi086 ceph-mon[29229]: pgmap v762: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:03 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:03 smithi086 ceph-mon[29229]: osdmap e266: 12 total, 3 up, 11 in 2024-08-06T18:00:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:03 smithi117 ceph-mon[32835]: pgmap v762: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:03 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2194301599' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:03 smithi117 ceph-mon[32835]: osdmap e266: 12 total, 3 up, 11 in 2024-08-06T18:00:03.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:03.776 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:03.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:04.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:04.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:04.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:04.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:04.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:04.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:04.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:04.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:04.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:05.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:05.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:05.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:00:05.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:05.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:05.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:05.644 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:05 smithi086 ceph-mon[29229]: pgmap v764: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:05.644 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:05.644 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:05.645 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:05.645 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:05.645 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:05 smithi117 ceph-mon[32835]: pgmap v764: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:05.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:05 smithi107 ceph-mon[32381]: pgmap v764: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:05.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:05.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:06.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:06.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:06.179 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:06.179 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:06.182 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:06.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:06.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:06.440 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:06.440 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:06.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:06.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:06.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:06.862 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:06.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:07.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:07.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:07.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:07.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:07 smithi117 ceph-mon[32835]: pgmap v765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:07.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:07 smithi117 ceph-mon[32835]: osdmap e267: 12 total, 3 up, 11 in 2024-08-06T18:00:07.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:07.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:07 smithi086 ceph-mon[29229]: pgmap v765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:07.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:07 smithi086 ceph-mon[29229]: osdmap e267: 12 total, 3 up, 11 in 2024-08-06T18:00:07.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:07 smithi107 ceph-mon[32381]: pgmap v765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 312 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:07 smithi107 ceph-mon[32381]: osdmap e267: 12 total, 3 up, 11 in 2024-08-06T18:00:07.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:07.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:08.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:08.155 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:08.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:00:08.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:08.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:08.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:08.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:08.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:08.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:08.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:08.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:08.881 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:08.882 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:08.882 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:08.898 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:08.899 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:08.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:08.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:08.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:08.916 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:08.917 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:08.918 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:09.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:09.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:09.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:09.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:09.420 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:09.421 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:09.421 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:09.421 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:09.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:09 smithi086 ceph-mon[29229]: pgmap v767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:09.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2597061667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:09.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.421 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:09.564 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:09 smithi107 ceph-mon[32381]: pgmap v767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:09.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2597061667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:09.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:09 smithi117 ceph-mon[32835]: pgmap v767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2597061667' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:09.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:09.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:09.833 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:09.833 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:09.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:09.952 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:09.952 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:10.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:10.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:10.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:10.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:10.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:10.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:10.508 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:00:10.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:10.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:10.508 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":267,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:00:06.299505+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:00:10.509 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:10.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:10.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:10.634 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:10.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:10.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:11.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:11.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:11.261 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:11.263 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:11.264 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:11.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:11.323 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:00:11.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:11.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:11 smithi086 ceph-mon[29229]: pgmap v768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:11.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/55945986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:11.485 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:11.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:11.580 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:11 smithi107 ceph-mon[32381]: pgmap v768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:11.581 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/55945986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:11.581 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:11.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:11 smithi117 ceph-mon[32835]: pgmap v768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:11.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/55945986' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:11.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:11.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:11.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:11.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:12.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:12.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:12.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:12.387 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:12 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:12.388 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:12.388 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:12 smithi107 ceph-mon[32381]: osdmap e268: 12 total, 3 up, 11 in 2024-08-06T18:00:12.388 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:12.388 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:12.465 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:12 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:12.465 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:12.465 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:12 smithi086 ceph-mon[29229]: osdmap e268: 12 total, 3 up, 11 in 2024-08-06T18:00:12.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:12.524 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:12 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:12.580 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/426801761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:12.581 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:12 smithi117 ceph-mon[32835]: osdmap e268: 12 total, 3 up, 11 in 2024-08-06T18:00:12.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:12.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:12.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:13.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:13.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:13.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:00:13.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:13.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:13 smithi086 ceph-mon[29229]: pgmap v770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:13.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:13.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:13.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:13 smithi117 ceph-mon[32835]: pgmap v770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:13.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:13.656 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:13 smithi107 ceph-mon[32381]: pgmap v770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:13.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:13.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:13.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:14.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:14.105 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:14.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:14.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:14.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:14.458 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:14.458 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:14.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:00:14.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:14.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:14.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:14.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:14.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:14.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:00:14.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:14.988 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:14.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:00:15.045 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:15.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:15.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:15.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:15.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:15.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:15.462 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:15 smithi086 ceph-mon[29229]: pgmap v771: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:15.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:15.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:15.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:15.531 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:15 smithi107 ceph-mon[32381]: pgmap v771: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:15.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:15 smithi117 ceph-mon[32835]: pgmap v771: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:15.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:15.765 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:15.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:15.795 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:15.797 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:15.797 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:15.802 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:15.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:16.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:16.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:16.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:16.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:16.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:16.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:16.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:16.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:17.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:17.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:17.302 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:17.302 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:17.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:17.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:17 smithi086 ceph-mon[29229]: pgmap v772: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:17 smithi086 ceph-mon[29229]: osdmap e269: 12 total, 3 up, 11 in 2024-08-06T18:00:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:17 smithi117 ceph-mon[32835]: pgmap v772: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:17 smithi117 ceph-mon[32835]: osdmap e269: 12 total, 3 up, 11 in 2024-08-06T18:00:17.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:17 smithi107 ceph-mon[32381]: pgmap v772: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:17 smithi107 ceph-mon[32381]: osdmap e269: 12 total, 3 up, 11 in 2024-08-06T18:00:17.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:17.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:18.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:18.143 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:18.144 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:18.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:18.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:18.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:18.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:18.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:18.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:18.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:19.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:19.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:19.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:19.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:19 smithi107 ceph-mon[32381]: pgmap v774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:19.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:19.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:19 smithi086 ceph-mon[29229]: pgmap v774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:19.588 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:19.588 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:19 smithi117 ceph-mon[32835]: pgmap v774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:19.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:19.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:19.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:19.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:19.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:19.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:19.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:19.962 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:19.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:20.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:20.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:20.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:20.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:20.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/447988286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:20.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.455 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/447988286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:20.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.561 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:20.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/447988286' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:20.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.585 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:20.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:20.804 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:20.804 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:20.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:20.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:20.977 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:20.977 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:21.022 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:21.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:21.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:21.237 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:21.237 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:21.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:21.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:21.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:21.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:21 smithi086 ceph-mon[29229]: pgmap v775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:21 smithi117 ceph-mon[32835]: pgmap v775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:21.675 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:21.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:00:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:21 smithi107 ceph-mon[32381]: pgmap v775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:21.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:21.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:22.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:22.024 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:22.025 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:22.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:22.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:22.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:22.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:22.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:22.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:22.579 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:22.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:22.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:22.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:00:22.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:22.953 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:22.953 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:22.980 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:22.980 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:23.017 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:00:23.017 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":269,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:00:16.301971+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:00:23.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:23.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:23.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:23.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:23.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:23.319 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:00:23.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:23.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:23.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:23.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:23 smithi117 ceph-mon[32835]: pgmap v776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:23.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/120961403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:23.610 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:00:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:23.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:23.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:23 smithi086 ceph-mon[29229]: pgmap v776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:23.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/120961403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:23.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:00:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:23 smithi107 ceph-mon[32381]: pgmap v776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/120961403' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:00:23.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:23.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:23.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:24.003 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:24.003 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:24.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:24.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:24.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:24.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:24.530 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:24.530 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:24.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:24.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:00:24.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:00:24.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:00:24.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:00:24.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: osdmap e270: 12 total, 3 up, 11 in 2024-08-06T18:00:24.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:24 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:00:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: osdmap e270: 12 total, 3 up, 11 in 2024-08-06T18:00:24.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:24 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:00:24.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:24.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:24.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1519053576' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:00:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: osdmap e270: 12 total, 3 up, 11 in 2024-08-06T18:00:24.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:24 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:00:24.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:24.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:24.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:24.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:25.024 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:25.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:25.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:25.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:25.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:25.455 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:25.456 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:25.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:25 smithi107 ceph-mon[32381]: pgmap v778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:25.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:00:25.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:00:25.456 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:25 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:00:25.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:25.587 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:25.587 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:25.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:25 smithi086 ceph-mon[29229]: pgmap v778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:00:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:00:25.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:25 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:00:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:25 smithi117 ceph-mon[32835]: pgmap v778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:00:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:00:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:25 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:00:25.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:25.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:25.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:25.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:25.840 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:25.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:25.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:26.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:26.102 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:00:26.102 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:00:26.102 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:26.102 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:26.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:26.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:26.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:26.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:26.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:26.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:26.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:26.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:26.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:00:26.933 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:26.933 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:27.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:00:27.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:27.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:27.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:27.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:00:27.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:27 smithi117 ceph-mon[32835]: pgmap v779: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:27.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:27 smithi117 ceph-mon[32835]: osdmap e271: 12 total, 3 up, 11 in 2024-08-06T18:00:27.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:27.464 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:27 smithi086 ceph-mon[29229]: pgmap v779: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:27.465 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:27 smithi086 ceph-mon[29229]: osdmap e271: 12 total, 3 up, 11 in 2024-08-06T18:00:27.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:27.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:27 smithi107 ceph-mon[32381]: pgmap v779: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:27.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:27 smithi107 ceph-mon[32381]: osdmap e271: 12 total, 3 up, 11 in 2024-08-06T18:00:27.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:27.774 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:27.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:27.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:27.996 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:27.997 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:28.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:28.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:28.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:28.294 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:28.294 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:28.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:28.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:28.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:28.710 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:28.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:28.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:28.949 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:28.950 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:29.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:29.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:29.208 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:29.209 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:29.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:29.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:29.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:29 smithi107 ceph-mon[32381]: pgmap v781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:29.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:29.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:29.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:29.615 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:29 smithi117 ceph-mon[32835]: pgmap v781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:29.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:29 smithi086 ceph-mon[29229]: pgmap v781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:29.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:29.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:29.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:30.072 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:30.073 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:30.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:30.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:30.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:30.282 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:30.283 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:30.283 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:30.283 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:30.283 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:30.300 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:30.301 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:30.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:30.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:30.425 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/86965985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:30.425 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:30.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:30.549 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/86965985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:30.549 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/86965985' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:30.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:30.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:30.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:30.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:30.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:31.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:31.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:31.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:31.374 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:31 smithi086 ceph-mon[29229]: pgmap v782: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:31.374 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.374 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:31 smithi107 ceph-mon[32381]: pgmap v782: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:31 smithi117 ceph-mon[32835]: pgmap v782: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:31.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:31.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:31.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:32.104 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:00:32.105 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":271,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:00:26.304555+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:00:32.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:32.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:32.326 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:00:32.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:32.411 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:32.411 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3973541493' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:32.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:32.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:32.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3973541493' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:32.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3973541493' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:32.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:32.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:33.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:33.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:33.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:33.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:33.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:33.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:33.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:33 smithi086 ceph-mon[29229]: pgmap v783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:33.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:33 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:33.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:33.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:33 smithi086 ceph-mon[29229]: osdmap e272: 12 total, 3 up, 11 in 2024-08-06T18:00:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:33 smithi117 ceph-mon[32835]: pgmap v783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:33 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:33 smithi117 ceph-mon[32835]: osdmap e272: 12 total, 3 up, 11 in 2024-08-06T18:00:33.629 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:33.629 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:33.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:33 smithi107 ceph-mon[32381]: pgmap v783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:33.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:33 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:33.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3174299084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:33.629 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:33 smithi107 ceph-mon[32381]: osdmap e272: 12 total, 3 up, 11 in 2024-08-06T18:00:33.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:33.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:34.010 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:34.010 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:34.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:34.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:34.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:34.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:34.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:34.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:34.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:34.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:34.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:34.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:34.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:34.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:35.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:35.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:35.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:00:35.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:35.396 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:35.396 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:35.396 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:35 smithi086 ceph-mon[29229]: pgmap v785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:35.410 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:00:35.411 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:00:35.411 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:00:35.411 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:00:35.411 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:00:35.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:35.537 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:35 smithi107 ceph-mon[32381]: pgmap v785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:35 smithi117 ceph-mon[32835]: pgmap v785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:35.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:35.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:35.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:35.931 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:35.931 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:35.931 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:35.932 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:36.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:36.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:00:36.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:36.198 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:36.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:36.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:36.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:36.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:36.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:00:36.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:36.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:36.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:00:36.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:36.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:37.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:37.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:37.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:37.469 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:37.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:37.546 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:37 smithi117 ceph-mon[32835]: pgmap v786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:37.546 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:37 smithi117 ceph-mon[32835]: osdmap e273: 12 total, 3 up, 11 in 2024-08-06T18:00:37.561 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:37 smithi086 ceph-mon[29229]: pgmap v786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:37.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:37 smithi086 ceph-mon[29229]: osdmap e273: 12 total, 3 up, 11 in 2024-08-06T18:00:37.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:37.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:37 smithi107 ceph-mon[32381]: pgmap v786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:37.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:37 smithi107 ceph-mon[32381]: osdmap e273: 12 total, 3 up, 11 in 2024-08-06T18:00:37.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:37.812 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:37.812 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:37.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:37.968 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:37.968 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:37.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:37.984 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:37.985 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:37.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:37.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:37.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:38.002 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:38.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:38.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:38.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:38.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:38.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:38.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:38.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:38.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:38.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2511616361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:38.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:38.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:38.618 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2511616361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:38.618 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.618 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:38.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2511616361' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:38.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:38.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:38.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:39.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:39.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:39.332 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:39.333 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:39.367 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:39.367 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:39.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:39.422 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:39 smithi107 ceph-mon[32381]: pgmap v788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:39.423 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:39.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:39.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:39 smithi086 ceph-mon[29229]: pgmap v788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:39.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:39 smithi117 ceph-mon[32835]: pgmap v788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:39.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:39.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:39.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:39.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:39.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:40.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:40.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:40.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:40.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:40.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:40.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:40.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:40.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:40.991 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:41.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:41.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:41.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:41.320 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:41.320 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:41.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:00:41.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:41.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:41.571 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:41 smithi117 ceph-mon[32835]: pgmap v789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:41.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:41 smithi086 ceph-mon[29229]: pgmap v789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:41.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:41.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:41.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:41 smithi107 ceph-mon[32381]: pgmap v789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:41.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:41.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:42.053 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:00:42.053 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:00:42.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:42.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:42.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:42.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:42.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:42.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:42.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:00:42.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:43.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:43.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:43.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:43.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:43.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:43 smithi107 ceph-mon[32381]: pgmap v790: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:43.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:43.464 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:43.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:43.588 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:43 smithi086 ceph-mon[29229]: pgmap v790: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:43 smithi117 ceph-mon[32835]: pgmap v790: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:43.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:43.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:43.904 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:00:43.905 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":273,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:00:36.308006+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:50.120366+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:00:43.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:44.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:44.117 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:44.117 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:44.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:44.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:44.395 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1347452869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:44.406 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1347452869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:44.411 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:44.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:44.529 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1347452869' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:44.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:44.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:44.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:44.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:44.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:45.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:45.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:45.198 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:45.198 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:45.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:45.334 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:00:45.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:45.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:45 smithi107 ceph-mon[32381]: pgmap v791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:45.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:45.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:45.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:45 smithi086 ceph-mon[29229]: pgmap v791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:45 smithi117 ceph-mon[32835]: pgmap v791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:00:45.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:45.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:45.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:45.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:45.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:46.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:46.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:46 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:46 smithi086 ceph-mon[29229]: osdmap e274: 12 total, 3 up, 11 in 2024-08-06T18:00:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:46 smithi086 ceph-mon[29229]: osdmap e275: 12 total, 3 up, 11 in 2024-08-06T18:00:46.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:46.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:46.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:46.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:46 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:46.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:46.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:46 smithi107 ceph-mon[32381]: osdmap e274: 12 total, 3 up, 11 in 2024-08-06T18:00:46.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:46 smithi107 ceph-mon[32381]: osdmap e275: 12 total, 3 up, 11 in 2024-08-06T18:00:46.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:46 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:00:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4019430809' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:00:46.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:46 smithi117 ceph-mon[32835]: osdmap e274: 12 total, 3 up, 11 in 2024-08-06T18:00:46.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:46 smithi117 ceph-mon[32835]: osdmap e275: 12 total, 3 up, 11 in 2024-08-06T18:00:46.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:46.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:46.852 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:46.853 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:46.869 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:46.869 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:46.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:46.889 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:46.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:47.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:47.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:47.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:47.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:47.585 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:47.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:47.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:47 smithi086 ceph-mon[29229]: pgmap v793: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:47.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1864080397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:47.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.600 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:47.600 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:47 smithi117 ceph-mon[32835]: pgmap v793: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1864080397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:47 smithi107 ceph-mon[32381]: pgmap v793: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1864080397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:47.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:47.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:47.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:48.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:48.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:48.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:48.347 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:48.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:48.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:00:48.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:48.555 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:48.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:48.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:48.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:48.769 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:48.769 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:48.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:48.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:49.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:49.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:49.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:49.350 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:49.350 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:49.351 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:49.351 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:49.397 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:49.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:49.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:49 smithi086 ceph-mon[29229]: pgmap v795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:49.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:49 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:00:49.589 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:49 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:00:49.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:49 smithi117 ceph-mon[32835]: pgmap v795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:49 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:00:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:49 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:00:49.639 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:49.639 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:49.639 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:49 smithi107 ceph-mon[32381]: pgmap v795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:49.639 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:49 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:00:49.639 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:49 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:00:49.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:49.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:49.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:49.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:50.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:50.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:50.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:50.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:50.437 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:50.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:50.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:00:50.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:50.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:00:50.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:50.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:51.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:51.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:00:51.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:51.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:51.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:51.509 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:51 smithi117 ceph-mon[32835]: pgmap v796: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:51 smithi086 ceph-mon[29229]: pgmap v796: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:51.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:51 smithi107 ceph-mon[32381]: pgmap v796: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:51.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:51.758 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:51.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:52.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:52.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:52.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:52.296 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:52.296 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:52.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:52.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:52.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:52.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:52.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:52.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:52.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:52.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:52.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:52.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:52.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:52.984 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:53.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:53.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:53.356 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:00:53.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:53.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:53.540 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:53 smithi107 ceph-mon[32381]: pgmap v797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:53.540 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:53.540 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:53.540 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:53 smithi086 ceph-mon[29229]: pgmap v797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:53.625 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:53 smithi117 ceph-mon[32835]: pgmap v797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:53.625 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:00:53.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:53.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:53.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:54.005 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:54.006 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:54.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:54.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:54.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:54.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:54.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:54 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:54.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:54.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:54 smithi086 ceph-mon[29229]: osdmap e276: 12 total, 3 up, 11 in 2024-08-06T18:00:54.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:54 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:54.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:54 smithi117 ceph-mon[32835]: osdmap e276: 12 total, 3 up, 11 in 2024-08-06T18:00:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:54 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:00:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/820239779' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:00:54.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:54 smithi107 ceph-mon[32381]: osdmap e276: 12 total, 3 up, 11 in 2024-08-06T18:00:54.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:54.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:54.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:55.113 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:00:55.113 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:00:55.113 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:00:55.119 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:55.128 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:00:55.129 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:00:55.129 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:00:55.129 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:00:55.129 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:00:55.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:55.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:00:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:00:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:00:55.150 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:00:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:00:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:00:55.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:00:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:55.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:55.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:55.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:55.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:55 smithi086 ceph-mon[29229]: pgmap v799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:55.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2658378308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:55.617 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:55.617 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:55.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:55 smithi107 ceph-mon[32381]: pgmap v799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2658378308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:55.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:00:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:55 smithi117 ceph-mon[32835]: pgmap v799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2658378308' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:00:55.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:56.011 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:56.011 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:56.011 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:00:56.011 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:00:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:00:56.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:00:56.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:56.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:56.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:56.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:56.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:00:56.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:00:56.626 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:00:56.634 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:56 smithi117 ceph-mon[32835]: osdmap e277: 12 total, 3 up, 11 in 2024-08-06T18:00:56.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:56.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:56.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:00:56.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:00:56.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:56 smithi107 ceph-mon[32381]: osdmap e277: 12 total, 3 up, 11 in 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:00:56.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:56 smithi086 ceph-mon[29229]: osdmap e277: 12 total, 3 up, 11 in 2024-08-06T18:00:56.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:56.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:57.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:57.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:00:57.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:57.299 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:00:57.299 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":277,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:00:55.371551+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:55.752487+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:59:49.892829+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:00:57.384 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:57.384 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:57.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:57.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:57.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:57 smithi086 ceph-mon[29229]: pgmap v801: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:57.578 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2073923567' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:57 smithi117 ceph-mon[32835]: pgmap v801: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2073923567' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:57.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:57 smithi107 ceph-mon[32381]: pgmap v801: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2073923567' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:00:57.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:57.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:58.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:58.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:58.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:00:58.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:00:58.528 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:58.528 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:58.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:00:58.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:00:58.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:00:58.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:00:59.012 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:00:59.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:00:59.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:59.209 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:00:59.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:00:59.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:00:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:00:59.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:00:59.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:59.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:59.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:00:59 smithi086 ceph-mon[29229]: pgmap v802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:59.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:00:59.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:00:59.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:00:59 smithi117 ceph-mon[32835]: pgmap v802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:59.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:00:59.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:00:59 smithi107 ceph-mon[32381]: pgmap v802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:00:59.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:00:59.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:00:59.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:00:59.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:00.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:00.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:00:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:00.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:00.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:00.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:00.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:00.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:00.550 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:00 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:01:00.550 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:00 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:01:00.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:00.681 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:00 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:01:00.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:00 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:01:00.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:00 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:01:00.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:00 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:01:00.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:00.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:01.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:01.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:01.293 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:01.293 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:01.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:01.395 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:01.395 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:01.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:01.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:01 smithi086 ceph-mon[29229]: pgmap v803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:01.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:01 smithi086 ceph-mon[29229]: osdmap e278: 12 total, 3 up, 11 in 2024-08-06T18:01:01.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:01 smithi086 ceph-mon[29229]: osdmap e279: 12 total, 3 up, 11 in 2024-08-06T18:01:01.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:01.606 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:01 smithi117 ceph-mon[32835]: pgmap v803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:01 smithi117 ceph-mon[32835]: osdmap e278: 12 total, 3 up, 11 in 2024-08-06T18:01:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:01 smithi117 ceph-mon[32835]: osdmap e279: 12 total, 3 up, 11 in 2024-08-06T18:01:01.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:01 smithi107 ceph-mon[32381]: pgmap v803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:01 smithi107 ceph-mon[32381]: osdmap e278: 12 total, 3 up, 11 in 2024-08-06T18:01:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:01 smithi107 ceph-mon[32381]: osdmap e279: 12 total, 3 up, 11 in 2024-08-06T18:01:01.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:01.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:01.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:02.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:02.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:02.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:02.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:02.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:02 smithi107 ceph-mon[32381]: 2.1 deep-scrub starts 2024-08-06T18:01:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:02 smithi107 ceph-mon[32381]: 2.1 deep-scrub ok 2024-08-06T18:01:02.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:02.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:02 smithi086 ceph-mon[29229]: 2.1 deep-scrub starts 2024-08-06T18:01:02.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:02 smithi086 ceph-mon[29229]: 2.1 deep-scrub ok 2024-08-06T18:01:02.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:02.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:02.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:02.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:02 smithi117 ceph-mon[32835]: 2.1 deep-scrub starts 2024-08-06T18:01:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:02 smithi117 ceph-mon[32835]: 2.1 deep-scrub ok 2024-08-06T18:01:02.885 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:02.885 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:02.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:03.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:03.047 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:03.063 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:03.064 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:03.081 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:03.082 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:03.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:03.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:03.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:03.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:03.083 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:03.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:03.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:03.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:03.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:03.511 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:03 smithi086 ceph-mon[29229]: pgmap v806: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:03.512 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/890100786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:03.512 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:03.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:03.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:03 smithi117 ceph-mon[32835]: pgmap v806: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:03.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/890100786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:03.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:03 smithi107 ceph-mon[32381]: pgmap v806: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/890100786' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:03.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:03.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:03.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:03.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:03.938 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:03.938 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:04.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:04.040 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:04.040 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:04.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:04.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:04.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:04.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:04.408 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:01:04.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:04.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:04.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:04.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:04.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:04.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:04.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:04.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:05.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:05.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:05.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:05.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:05.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:05.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:05.519 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:05 smithi117 ceph-mon[32835]: pgmap v807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:05.519 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:05 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:05.519 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:05.520 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:05 smithi117 ceph-mon[32835]: osdmap e280: 12 total, 3 up, 11 in 2024-08-06T18:01:05.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:05.636 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:05 smithi086 ceph-mon[29229]: pgmap v807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:05.636 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:05 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:05.636 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:05.636 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:05 smithi086 ceph-mon[29229]: osdmap e280: 12 total, 3 up, 11 in 2024-08-06T18:01:05.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:05 smithi107 ceph-mon[32381]: pgmap v807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:05.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:05 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:05.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1363569853' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:05.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:05 smithi107 ceph-mon[32381]: osdmap e280: 12 total, 3 up, 11 in 2024-08-06T18:01:05.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:05.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:05.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:01:06.005 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:06.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:06.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:06.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:06.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:06.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:06.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:06.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:06.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:06 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:01:06.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:06 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:01:06.576 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:06 smithi086 ceph-mon[29229]: osdmap e281: 12 total, 3 up, 11 in 2024-08-06T18:01:06.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:06.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:01:06.650 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:06.682 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:06.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:06 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:01:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:06 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:01:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:06 smithi107 ceph-mon[32381]: osdmap e281: 12 total, 3 up, 11 in 2024-08-06T18:01:06.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:06 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:01:06.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:06 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:01:06.732 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:06 smithi117 ceph-mon[32835]: osdmap e281: 12 total, 3 up, 11 in 2024-08-06T18:01:06.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:06.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:06.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:06.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:06.993 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:01:07.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:07.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:07.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:07.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:07.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:07.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:07 smithi117 ceph-mon[32835]: pgmap v809: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:07.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:07 smithi107 ceph-mon[32381]: pgmap v809: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:07.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:07 smithi086 ceph-mon[29229]: pgmap v809: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:07.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:08.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:08.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:08.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:08.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:08.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:08.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:08.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:08.672 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:08.672 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:08.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:08.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:08.945 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:08.945 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":281,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:06.316107+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:08.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:09.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:09.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:09.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:09.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:09.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:09 smithi086 ceph-mon[29229]: pgmap v811: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:09.542 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2690975878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:09.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:09.677 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:09 smithi107 ceph-mon[32381]: pgmap v811: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:09.677 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2690975878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:09.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:09 smithi117 ceph-mon[32835]: pgmap v811: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2690975878' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:09.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:09.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:09.951 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:10.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:10.074 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:10.074 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:10.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:10.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:10.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:10.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:10.597 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:10.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:10.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:10.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:10.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:11.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:11.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:11.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:11.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:11 smithi117 ceph-mon[32835]: pgmap v812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:11.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:11.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:11 smithi107 ceph-mon[32381]: pgmap v812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:11.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:11.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:11 smithi086 ceph-mon[29229]: pgmap v812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:11.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:12.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:12.110 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:12.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:12.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:12.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:12.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:12.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:12.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:12.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:12.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:12.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:12.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:13.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:13.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:13.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:13.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:13.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:13 smithi107 ceph-mon[32381]: pgmap v813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:13.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:13.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:13.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:13.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:13 smithi086 ceph-mon[29229]: pgmap v813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:13.602 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:13.603 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:13.618 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:13.618 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:13.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:13.619 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:13.619 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:13 smithi117 ceph-mon[32835]: pgmap v813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:13.635 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:13.636 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:13.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:13.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:13.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:13.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:13.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:13.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:13.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:13.820 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:01:13.820 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:01:13.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:14.066 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:14.066 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:14.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:14.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:14.425 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:01:14.426 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:14.426 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:14.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:14.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3553615408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:14.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:14.745 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:14.745 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:14.745 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:14.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:14.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3553615408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:14.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.746 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:14.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3553615408' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:14.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:14.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:01:14.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:15.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:15.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:15.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:15.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:15.562 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:15.562 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:15.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:15.563 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:15 smithi086 ceph-mon[29229]: pgmap v814: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:15.563 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:15 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:15.563 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:15.563 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:15 smithi086 ceph-mon[29229]: osdmap e282: 12 total, 3 up, 11 in 2024-08-06T18:01:15.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:15.696 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:15.696 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:15.697 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:15.697 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:15 smithi107 ceph-mon[32381]: pgmap v814: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:15.697 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:15 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:15.697 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:15.697 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:15 smithi107 ceph-mon[32381]: osdmap e282: 12 total, 3 up, 11 in 2024-08-06T18:01:15.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:15 smithi117 ceph-mon[32835]: pgmap v814: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:15 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:15.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3052716825' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:15.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:15 smithi117 ceph-mon[32835]: osdmap e282: 12 total, 3 up, 11 in 2024-08-06T18:01:15.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:16.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:16.094 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:16.094 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:16.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:16.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:16.340 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:16.345 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:16.345 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:16.346 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:16.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:16.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:16.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:16.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:16.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:16.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:17.073 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:17.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:17.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:17.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:17.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:17 smithi107 ceph-mon[32381]: pgmap v816: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:17.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:17 smithi107 ceph-mon[32381]: osdmap e283: 12 total, 3 up, 11 in 2024-08-06T18:01:17.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:17.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:17 smithi086 ceph-mon[29229]: pgmap v816: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:17.549 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:17 smithi086 ceph-mon[29229]: osdmap e283: 12 total, 3 up, 11 in 2024-08-06T18:01:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:17 smithi117 ceph-mon[32835]: pgmap v816: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:17 smithi117 ceph-mon[32835]: osdmap e283: 12 total, 3 up, 11 in 2024-08-06T18:01:17.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:17.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:01:17.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:17.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:17.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:17.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:18.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:18.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:18.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:18.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:01:18.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:18.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:18.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:18.842 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:18.843 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:18.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:18.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:18.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:18.995 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:18.995 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":283,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:16.319030+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:19.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:19.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:19.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:19.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:19.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:19.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:19 smithi086 ceph-mon[29229]: pgmap v818: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:19.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1851390974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:19.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:19.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:19 smithi117 ceph-mon[32835]: pgmap v818: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1851390974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:19.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:19.686 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:19 smithi107 ceph-mon[32381]: pgmap v818: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:19.687 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1851390974' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:19.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:19.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:19.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:20.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:20.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:20.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:20.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:20.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:20.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:20.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:20.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:20.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:20.873 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:20.874 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:20.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:21.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:21.136 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:21.136 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:21.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:21.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:21.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:21 smithi107 ceph-mon[32381]: pgmap v819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:21.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:21.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:21 smithi117 ceph-mon[32835]: pgmap v819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:21 smithi086 ceph-mon[29229]: pgmap v819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:21.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:21.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:21.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:22.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:22.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:22.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:22.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:22.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:22.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:22.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:22.692 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:22.692 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:22.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:22.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:22.951 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:22.952 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:22.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:23.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:23.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:23.221 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:23.221 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:23.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:23.346 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:23.346 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:23.346 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:23.347 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:23.348 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:01:23.351 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:23.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:23.375 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:23.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:23.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:23.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:23 smithi086 ceph-mon[29229]: pgmap v820: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:23.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:23.532 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1232404559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:23.532 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:23.532 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:23 smithi117 ceph-mon[32835]: pgmap v820: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1232404559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:23.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:23 smithi107 ceph-mon[32381]: pgmap v820: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1232404559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:23.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:23.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:24.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:24.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:24.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:24.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:24.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:24.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:24.601 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:24.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:24.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:24.602 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: osdmap e284: 12 total, 3 up, 11 in 2024-08-06T18:01:24.602 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: osdmap e284: 12 total, 3 up, 11 in 2024-08-06T18:01:24.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:24.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1636813787' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: osdmap e284: 12 total, 3 up, 11 in 2024-08-06T18:01:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:24.751 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:01:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:01:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:01:24.752 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:01:24.752 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:01:24.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:24.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:25.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:25.204 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:25.205 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:25.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:25.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:25.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:25 smithi107 ceph-mon[32381]: pgmap v822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:25.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:25 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:25.503 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:25 smithi086 ceph-mon[29229]: pgmap v822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:25.504 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:25 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:25.504 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:25.504 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:25.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:25 smithi117 ceph-mon[32835]: pgmap v822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:25 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:25.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:25.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:25.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:25.930 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:25.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:01:26.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:26.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:01:26.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:26.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:26.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:26.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: osdmap e285: 12 total, 3 up, 11 in 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:26.608 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:26.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:26 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: osdmap e285: 12 total, 3 up, 11 in 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:26.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:26.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:26.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:26 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: osdmap e285: 12 total, 3 up, 11 in 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:26.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:26.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:26 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:26.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:26.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:26.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:26.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:26.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:27.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:27.146 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:27.146 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:27.180 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:27.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:01:27.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:27.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:27.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:27.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:27.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:27.837 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:27.871 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:27.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:27.937 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:27.937 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:28.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:28.069 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:28.069 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:28.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:28.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:28.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:28.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:28.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:28.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:28.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:28.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:28.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:28.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:28.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:28.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:28.998 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:28 smithi107 ceph-mon[32381]: pgmap v825: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:29.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:28 smithi086 ceph-mon[29229]: pgmap v825: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:29.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:28 smithi117 ceph-mon[32835]: pgmap v825: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:29.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:29.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:29.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:29.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:29.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:29.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:29.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:29.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:29.939 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:29.939 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:30.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:30.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:30.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:30.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:30.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:30.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:30.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:30.861 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:30.861 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:30.868 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:30.868 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":285,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:26.321472+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:30.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:31.011 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:30 smithi117 ceph-mon[32835]: pgmap v826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:31.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:30 smithi086 ceph-mon[29229]: pgmap v826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:31.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:31.117 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:31.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:30 smithi107 ceph-mon[32381]: pgmap v826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:31.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:31.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:31.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:31.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:31.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:31.862 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:31.862 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:31.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:32.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:32.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3342100630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3342100630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:32.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3342100630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:32.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:32.330 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:01:32.330 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:01:32.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:167,from:0,reservation_nonce:0)", 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123194+0000", 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 471.20813593600002, 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000208364, 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123194+0000", 2024-08-06T18:01:32.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.7299999999999995e-07 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123197+0000", 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7089999999999998e-06 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123207+0000", 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6819999999999998e-06 2024-08-06T18:01:32.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123212+0000", 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6380000000000001e-06 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123253+0000", 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0308000000000001e-05 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123272+0000", 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9191e-05 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123402+0000", 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013016300000000001 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:01:32.335 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:53:41.123882+0000", 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 471.20744793799997, 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8758999999999996e-05, 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T18:01:32.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123882+0000", 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7800000000000002e-07 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123885+0000", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9720000000000001e-06 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123892+0000", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2119999999999997e-06 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123896+0000", 2024-08-06T18:01:32.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8580000000000002e-06 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123933+0000", 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6906000000000002e-05 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123941+0000", 2024-08-06T18:01:32.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8709999999999997e-06 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:53:41.123960+0000", 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9462000000000001e-05 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.340 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:240,from:0,reservation_nonce:0)", 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 193.594944043, 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019324500000000001, 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2099999999999995e-07 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736390+0000", 2024-08-06T18:01:32.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3009999999999998e-06 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736400+0000", 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.065e-05 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736408+0000", 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6809999999999998e-06 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736448+0000", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0349999999999998e-05 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736461+0000", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3015e-05 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736579+0000", 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117427 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.344 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 193.59423157399999, 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5890999999999996e-05, 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.345 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.53e-07 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737101+0000", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8830000000000002e-06 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737109+0000", 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6899999999999992e-06 2024-08-06T18:01:32.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737113+0000", 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9489999999999998e-06 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737147+0000", 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4094000000000003e-05 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737154+0000", 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1760000000000004e-06 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737174+0000", 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9746e-05 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:259,from:0,reservation_nonce:0)", 2024-08-06T18:01:32.349 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 111.559657481, 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000226534, 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771673+0000", 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0399999999999995e-07 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771676+0000", 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7480000000000002e-06 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771685+0000", 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1759999999999999e-06 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771690+0000", 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2950000000000001e-06 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771734+0000", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218000000000003e-05 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771750+0000", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6150999999999998e-05 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771899+0000", 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014824200000000001 2024-08-06T18:01:32.353 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 111.55893910099999, 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8885000000000002e-05, 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:01:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0699999999999998e-07 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772394+0000", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5969999999999999e-06 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772400+0000", 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9539999999999998e-06 2024-08-06T18:01:32.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772404+0000", 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6399999999999999e-06 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772435+0000", 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0793999999999997e-05 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:01:32.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772441+0000", 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.224e-06 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772459+0000", 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8269000000000001e-05 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:01:32.358 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:01:32.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:32.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:32.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:32.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:32.868 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:01:33.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:33.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:32 smithi086 ceph-mon[29229]: pgmap v827: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:33.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:33.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:01:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:32 smithi117 ceph-mon[32835]: pgmap v827: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:33.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:33.145 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:32 smithi107 ceph-mon[32381]: pgmap v827: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:33.146 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:33.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:33.378 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:33.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:33.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:33.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:33.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:33.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:34.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:34.117 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:34.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:33 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:01:34.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:34.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:33 smithi086 ceph-mon[29229]: osdmap e286: 12 total, 3 up, 11 in 2024-08-06T18:01:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:33 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:01:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:33 smithi117 ceph-mon[32835]: osdmap e286: 12 total, 3 up, 11 in 2024-08-06T18:01:34.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:34.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:33 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:01:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1468626696' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:33 smithi107 ceph-mon[32381]: osdmap e286: 12 total, 3 up, 11 in 2024-08-06T18:01:34.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:34.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:34.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:34.518 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:34.518 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:34.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:34.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:34.778 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:34.778 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:34.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:34.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:34.854 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:34.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:34.876 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:34.877 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:34.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:34.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:34.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:34.895 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:34.896 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:34.896 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:34.896 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:34.896 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:34.897 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:34.897 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:34.898 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:34.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:35.013 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:35.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:35.092 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:34 smithi107 ceph-mon[32381]: pgmap v829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:35.092 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2214787064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:34 smithi117 ceph-mon[32835]: pgmap v829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2214787064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:35.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:35.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:34 smithi086 ceph-mon[29229]: pgmap v829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:35.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2214787064' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:35.204 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:35.204 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:35.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:35.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:35.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:35.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:35.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:35.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:35.620 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:35.645 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:35.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:35.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:35.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:01:35.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:35.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:01:35.878 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:35.878 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:35.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:36.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.073 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:36.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:36.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:36.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:36.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:36.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:36.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:36.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:37.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:37.172 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:37.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:37.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:37.379 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:37.379 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":287,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:36.324349+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:37.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:37.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:37 smithi107 ceph-mon[32381]: pgmap v830: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:37.449 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:37 smithi107 ceph-mon[32381]: osdmap e287: 12 total, 3 up, 11 in 2024-08-06T18:01:37.449 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:37.450 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:37.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:37.601 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:37.601 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:37.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:37 smithi086 ceph-mon[29229]: pgmap v830: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:37.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:37 smithi086 ceph-mon[29229]: osdmap e287: 12 total, 3 up, 11 in 2024-08-06T18:01:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:37 smithi117 ceph-mon[32835]: pgmap v830: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:37 smithi117 ceph-mon[32835]: osdmap e287: 12 total, 3 up, 11 in 2024-08-06T18:01:37.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:01:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:37.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:37.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:37.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:38.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:38.246 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:38.343 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:38.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:01:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/641980346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:38.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:38.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:38.595 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:38.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/641980346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:38.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:38.671 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/641980346' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:38.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:38.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:39.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:39.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:39.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:39.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:39.336 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:39.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:39.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:39.600 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:39 smithi086 ceph-mon[29229]: pgmap v832: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:39.605 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:39.618 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:39 smithi117 ceph-mon[32835]: pgmap v832: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:39.710 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:39 smithi107 ceph-mon[32381]: pgmap v832: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:39.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:39.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:39.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:40.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:40.163 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:40.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:40.343 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:01:40.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:40.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:40.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:40.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:40.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:01:40.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:40.843 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:40.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:01:40.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:41.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:41.091 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:41.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:41.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:41.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:41.333 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:41.334 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:41.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: osdmap e288: 12 total, 3 up, 11 in 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:41.473 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:41.474 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:41.474 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:41.474 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:41.474 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:41 smithi086 ceph-mon[29229]: osdmap e289: 12 total, 3 up, 11 in 2024-08-06T18:01:41.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:41.607 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:41.607 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: osdmap e288: 12 total, 3 up, 11 in 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:41.608 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:41.609 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:41.609 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:41.609 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:41 smithi107 ceph-mon[32381]: osdmap e289: 12 total, 3 up, 11 in 2024-08-06T18:01:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2284586149' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:01:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: osdmap e288: 12 total, 3 up, 11 in 2024-08-06T18:01:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:01:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:41 smithi117 ceph-mon[32835]: osdmap e289: 12 total, 3 up, 11 in 2024-08-06T18:01:41.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:41.763 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:41.763 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:41.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:41.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:42.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:42.140 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:42.141 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:42.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:42.160 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:42.161 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:42.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:42.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:42.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:42.178 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:42.182 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:42.182 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:42.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:42.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:42.185 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:42.186 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:42.186 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:42.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:42.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:42.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:42.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:42.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:42.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2024794280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:42.572 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2024794280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:42.572 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:42.572 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2024794280' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:42.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:42.768 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:42.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:42.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:43.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:43.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:43.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:43.201 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:43.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:43.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:43.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:43 smithi107 ceph-mon[32381]: pgmap v836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:43.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.454 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.483 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:43.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:01:43.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:43.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:43 smithi086 ceph-mon[29229]: pgmap v836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:01:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:43 smithi117 ceph-mon[32835]: pgmap v836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:43.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:43.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:43.777 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:43.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:43.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:44.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:44.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:44.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:44.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:44.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:44.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:44.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:44.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:45.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:45.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:45.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:01:45.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:45.338 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:45.338 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":289,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:41.325909+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:45.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:45.431 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:45.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:45 smithi086 ceph-mon[29229]: pgmap v837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/802577978' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:45 smithi117 ceph-mon[32835]: pgmap v837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/802577978' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:45.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:45 smithi107 ceph-mon[32381]: pgmap v837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 313 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/802577978' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:45.833 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:45.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:46.001 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:46.002 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:46.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:46.112 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:46.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:01:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:46.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:46.308 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:46.308 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:46.361 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:46.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:46.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:46.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:46.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:46.785 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:46.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:47.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:47.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:47.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:47.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:47.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:01:47.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:47.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:47.592 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:47 smithi086 ceph-mon[29229]: pgmap v838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:47.592 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:47 smithi117 ceph-mon[32835]: pgmap v838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:47.644 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:47 smithi107 ceph-mon[32381]: pgmap v838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:47.644 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:01:47.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:47.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:47.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:48.000 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:48.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:48.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:48.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:48.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:48.525 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:48 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:48.525 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:48.525 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:48 smithi117 ceph-mon[32835]: osdmap e290: 12 total, 3 up, 11 in 2024-08-06T18:01:48.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:48 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:48.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:48.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:48 smithi086 ceph-mon[29229]: osdmap e290: 12 total, 3 up, 11 in 2024-08-06T18:01:48.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:48.638 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:48 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:48.638 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1259604455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:01:48.638 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:48 smithi107 ceph-mon[32381]: osdmap e290: 12 total, 3 up, 11 in 2024-08-06T18:01:48.664 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:48.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:01:48.770 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:48.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:48.909 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:49.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:49.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:49.213 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:49.214 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:49.230 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:49.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:49.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:49.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:49.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:49.248 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:49.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:49.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:49.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:49.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:49.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:49.589 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:49.590 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:49.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:49 smithi086 ceph-mon[29229]: pgmap v840: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:49.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/89029628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:49 smithi117 ceph-mon[32835]: pgmap v840: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/89029628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:49.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:49.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:49 smithi107 ceph-mon[32381]: pgmap v840: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:49.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/89029628' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:49.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:49.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:50.003 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:50.003 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:50.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:50.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:50.063 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:50.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:50.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:50.261 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:50.261 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:50.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:01:50.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:50.425 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.425 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.425 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:50.500 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:50.500 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.500 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.500 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:50.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.565 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:50.666 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:01:50.666 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:01:50.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:50.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:50.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:50.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:50.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:51.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:51.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:51.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:51.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:01:51.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:51.494 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:51 smithi107 ceph-mon[32381]: pgmap v841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:51.494 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:51 smithi107 ceph-mon[32381]: osdmap e291: 12 total, 3 up, 11 in 2024-08-06T18:01:51.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:51 smithi086 ceph-mon[29229]: pgmap v841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:51.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:51 smithi086 ceph-mon[29229]: osdmap e291: 12 total, 3 up, 11 in 2024-08-06T18:01:51.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:51.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:51.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:51 smithi117 ceph-mon[32835]: pgmap v841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:51 smithi117 ceph-mon[32835]: osdmap e291: 12 total, 3 up, 11 in 2024-08-06T18:01:51.665 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:51.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:51.773 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:51.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:51.926 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:51.926 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":291,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:51.328542+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:52.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:01:52.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:52.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:52.272 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:52.272 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:52.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:52.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:52.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3568413305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:52.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:52.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:52.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3568413305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:52.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3568413305' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:52.842 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:52.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:53.094 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:53.094 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:53.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:53.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:53.317 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:01:53.318 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:01:53.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:53.402 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:53 smithi117 ceph-mon[32835]: pgmap v843: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:53.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:53 smithi086 ceph-mon[29229]: pgmap v843: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:53.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:53.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:53.638 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:53 smithi107 ceph-mon[32381]: pgmap v843: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:53.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:53.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:53.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:54.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:54.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:01:54.079 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:54.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:01:54.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:54.320 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:54.337 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:01:54.338 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:01:54.384 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:01:54.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:54.478 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:54.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:54.589 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:54.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:54.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:01:54.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:55.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:55.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:55.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:55.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:55 smithi117 ceph-mon[32835]: pgmap v844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:55 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:55.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:55 smithi117 ceph-mon[32835]: osdmap e292: 12 total, 3 up, 11 in 2024-08-06T18:01:55.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:55.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:55 smithi107 ceph-mon[32381]: pgmap v844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:55.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:55 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:55.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:55.659 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:55 smithi107 ceph-mon[32381]: osdmap e292: 12 total, 3 up, 11 in 2024-08-06T18:01:55.675 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:55.675 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:55.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:55 smithi086 ceph-mon[29229]: pgmap v844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:55.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:55 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:01:55.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1050592090' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:01:55.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:55 smithi086 ceph-mon[29229]: osdmap e292: 12 total, 3 up, 11 in 2024-08-06T18:01:55.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:01:55.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:01:55.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:55.990 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:01:55.990 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:01:56.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:01:56.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:56.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:56.369 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:01:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:01:56.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:01:56.374 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:01:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:01:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:01:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:01:56.374 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:01:56.392 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:01:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:01:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:01:56.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:01:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:01:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:01:56.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:01:56.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:56.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:01:56.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:56.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:56.836 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:56.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:56.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:57.029 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:01:57.029 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:01:57.059 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:57.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:57.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:01:57.317 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:01:57.318 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:01:57.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: pgmap v846: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: osdmap e293: 12 total, 3 up, 11 in 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/147997530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:01:57.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:57.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: pgmap v846: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: osdmap e293: 12 total, 3 up, 11 in 2024-08-06T18:01:57.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/147997530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:57.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.617 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: pgmap v846: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 317 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: osdmap e293: 12 total, 3 up, 11 in 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/147997530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:01:57.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:57.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:57.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:01:57.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:58.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:58.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:58.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:58.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:58.527 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:01:58.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:01:58.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:58.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:01:58.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:58.945 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:01:58.945 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":293,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:01:56.330392+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:01:58.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:01:59.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:01:59.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:59.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:01:59.268 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:01:59.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:01:59.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:01:59.384 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:01:59.385 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:01:59.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:59 smithi086 ceph-mon[29229]: pgmap v848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:59.385 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:01:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3626669736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:59.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:01:59.514 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:59 smithi107 ceph-mon[32381]: pgmap v848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:59.515 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:01:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3626669736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:59.559 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:01:59.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:59 smithi117 ceph-mon[32835]: pgmap v848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:01:59.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:01:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3626669736' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:01:59.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:01:59.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:01:59.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:00.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:00.043 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:00.043 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:01:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:00.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:00.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:00.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:00.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:01:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:00.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:00.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:00.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:00.438 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:00.438 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:00.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:00.602 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:00.603 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:00.674 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:00.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:00.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:00.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:01.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:01.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:01.260 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:01.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:01.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:01 smithi117 ceph-mon[32835]: pgmap v849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:01.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:01 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:02:01.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:01 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:02:01.450 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:01 smithi086 ceph-mon[29229]: pgmap v849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:01.451 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:01 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:02:01.451 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:01 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:02:01.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:01.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:01.702 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:01.702 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:01.703 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:01 smithi107 ceph-mon[32381]: pgmap v849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:01.703 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:01 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:02:01.703 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:01 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:02:01.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:01.764 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:01.764 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:01.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:01.956 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:01.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:02:02.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:02.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:02.206 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:02:02.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:02.341 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:02.341 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:02.342 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:02:02.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:02.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:02.560 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:02.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:02.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:02.648 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:02.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:02.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:02.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:03.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:03.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:03.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:03.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:03.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:03 smithi086 ceph-mon[29229]: pgmap v850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:03.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:03 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:03.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:03.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:03 smithi086 ceph-mon[29229]: osdmap e294: 12 total, 3 up, 11 in 2024-08-06T18:02:03.588 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:03.588 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:03.606 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:03.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:03.607 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:03.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:03 smithi117 ceph-mon[32835]: pgmap v850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:03 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:03.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:03 smithi117 ceph-mon[32835]: osdmap e294: 12 total, 3 up, 11 in 2024-08-06T18:02:03.624 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:03.628 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:03.628 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:03.629 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:03.635 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:03 smithi107 ceph-mon[32381]: pgmap v850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:03.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:03 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:03.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4211531472' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:03.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:03 smithi107 ceph-mon[32381]: osdmap e294: 12 total, 3 up, 11 in 2024-08-06T18:02:03.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:03.845 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:03.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:04.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:04.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:04.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:04.334 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:04.334 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:04.363 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:04.363 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:04.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:04.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2685949702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:04.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.461 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:04.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:02:04.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:04.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2685949702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:04.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.590 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2685949702' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:04.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:04.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:04.736 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:04.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:04.847 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:04.847 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:04.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:05.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:05.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:05.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:05.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:05.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:02:05.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:05.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:05.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:05 smithi117 ceph-mon[32835]: pgmap v852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:05.415 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:05.416 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:05 smithi086 ceph-mon[29229]: pgmap v852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:05.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:05.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:05.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:05.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:05 smithi107 ceph-mon[32381]: pgmap v852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:05.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:05.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:05.698 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:05.699 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:05.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:05.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:05.970 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:06.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:06.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:06.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:06.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:06.126 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":294,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:02.338726+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:55.287514+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:06.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:06.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:06.298 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:06.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:02:06.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:06.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2973481741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:06 smithi086 ceph-mon[29229]: osdmap e295: 12 total, 3 up, 11 in 2024-08-06T18:02:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:06.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2973481741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:06.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:06 smithi117 ceph-mon[32835]: osdmap e295: 12 total, 3 up, 11 in 2024-08-06T18:02:06.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:06.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2973481741' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:06 smithi107 ceph-mon[32381]: osdmap e295: 12 total, 3 up, 11 in 2024-08-06T18:02:06.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:06.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:06.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:02:06.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:06.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:06.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:07.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:07.114 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:07.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:07.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:07.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:07.537 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:07 smithi117 ceph-mon[32835]: pgmap v853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:07.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:07.614 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:07 smithi086 ceph-mon[29229]: pgmap v853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:07 smithi107 ceph-mon[32381]: pgmap v853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:07.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:07.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:07.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:08.078 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:08.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:08.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:08.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:08.419 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:08.419 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:08.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:08.622 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:08.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:08.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:08.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:09.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:09.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:02:09.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:09.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:09.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:09.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:09.368 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:02:09.426 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:09.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:09.547 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:09 smithi117 ceph-mon[32835]: pgmap v855: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:09.547 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:09.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:09.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:09 smithi086 ceph-mon[29229]: pgmap v855: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:09.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:09.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:09 smithi107 ceph-mon[32381]: pgmap v855: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:09.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:09.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:09.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:09.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:09.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:10.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:10.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:10.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:10.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:10.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:10.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:10.545 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:10 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:10.545 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:10.545 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:10 smithi117 ceph-mon[32835]: osdmap e296: 12 total, 3 up, 11 in 2024-08-06T18:02:10.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:10 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:10.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:10.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:10 smithi086 ceph-mon[29229]: osdmap e296: 12 total, 3 up, 11 in 2024-08-06T18:02:10.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:10.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:10 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3328823814' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:10.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:10 smithi107 ceph-mon[32381]: osdmap e296: 12 total, 3 up, 11 in 2024-08-06T18:02:10.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:10.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:10.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:10.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:10.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:10.826 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:10.827 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:10.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:10.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:10.828 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:10.828 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:10.828 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:10.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:10.998 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:11.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:11.067 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:11.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:11.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:11.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:11.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:11.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:11.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:11.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:11.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:11 smithi107 ceph-mon[32381]: pgmap v857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:11.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1229260487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:11.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:11 smithi107 ceph-mon[32381]: osdmap e297: 12 total, 3 up, 11 in 2024-08-06T18:02:11.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:11.595 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:11.595 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:11 smithi117 ceph-mon[32835]: pgmap v857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1229260487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:11 smithi117 ceph-mon[32835]: osdmap e297: 12 total, 3 up, 11 in 2024-08-06T18:02:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:11.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:02:11.721 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:11.721 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:11.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:11 smithi086 ceph-mon[29229]: pgmap v857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:11.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1229260487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:11.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:11.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:11 smithi086 ceph-mon[29229]: osdmap e297: 12 total, 3 up, 11 in 2024-08-06T18:02:11.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:11.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:11.948 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:11.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:11.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:12.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:12.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:12.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:12.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:12.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:12.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:02:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:12.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:12.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:12.769 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:12.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:12.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:13.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:13.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:13.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:13.396 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:13.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:13.497 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:13.509 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:13.509 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":298,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:13.389265+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:01:05.823785+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:00:59.974101+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:13.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:13.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:02:13.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:13 smithi117 ceph-mon[32835]: pgmap v859: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:13 smithi107 ceph-mon[32381]: pgmap v859: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:13.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:13.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:13 smithi086 ceph-mon[29229]: pgmap v859: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:13.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:13.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:14.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:14.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:14.200 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:14.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:02:14.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:14.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:14.397 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:14.398 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:14.448 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:14.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:14.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:14 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:02:14.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:14 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:02:14.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:14 smithi117 ceph-mon[32835]: osdmap e298: 12 total, 3 up, 11 in 2024-08-06T18:02:14.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3014642268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:14.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:14.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:14 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:02:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:14 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:02:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:14 smithi107 ceph-mon[32381]: osdmap e298: 12 total, 3 up, 11 in 2024-08-06T18:02:14.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3014642268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:14.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:14.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:14 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:02:14.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:14 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:02:14.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:14 smithi086 ceph-mon[29229]: osdmap e298: 12 total, 3 up, 11 in 2024-08-06T18:02:14.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3014642268' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:14.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:15.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:15.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:15.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:15.316 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:15.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:15.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:15 smithi086 ceph-mon[29229]: pgmap v861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:15.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:15 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:02:15.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:15 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:02:15.522 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:15 smithi117 ceph-mon[32835]: pgmap v861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:15.522 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:15 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:02:15.523 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:15 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:02:15.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:15 smithi107 ceph-mon[32381]: pgmap v861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:15 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:02:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:15 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:02:15.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:15.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:15.967 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:16.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:16.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:16.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:16.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:16.214 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:16.214 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:16.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:16.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:16.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:16.566 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:16 smithi117 ceph-mon[32835]: osdmap e299: 12 total, 3 up, 11 in 2024-08-06T18:02:16.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:16.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:02:16.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:16 smithi086 ceph-mon[29229]: osdmap e299: 12 total, 3 up, 11 in 2024-08-06T18:02:16.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:16.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:16.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:16.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:16 smithi107 ceph-mon[32381]: osdmap e299: 12 total, 3 up, 11 in 2024-08-06T18:02:16.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:16.900 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:16.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:17.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:17.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:17.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:17.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:17.251 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:17.252 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:17.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:17.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:17.414 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:02:17.530 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:17.531 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:17.531 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:17.532 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:17.533 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:17.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:17.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:17.534 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:17.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:17.551 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:17.552 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:17.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:17.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:17.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:17.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:17.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:17.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:17 smithi117 ceph-mon[32835]: pgmap v863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:17.558 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:17.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:17 smithi107 ceph-mon[32381]: pgmap v863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:17.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:17 smithi086 ceph-mon[29229]: pgmap v863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:17.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:17.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:17.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:18.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:18.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:18.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:18.347 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:18.347 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:18.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:18.488 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: osdmap e300: 12 total, 3 up, 11 in 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1093444915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: osdmap e300: 12 total, 3 up, 11 in 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1093444915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:18.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2168522865' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: osdmap e300: 12 total, 3 up, 11 in 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1093444915' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:18.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:18.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:18.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:19.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:19.127 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:19.194 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:19.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:02:19.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:19.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:19.481 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:19.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:19.543 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:19 smithi107 ceph-mon[32381]: pgmap v865: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:19.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:19.677 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:19 smithi086 ceph-mon[29229]: pgmap v865: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:19.699 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:19 smithi117 ceph-mon[32835]: pgmap v865: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:19.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:19.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:19.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:19.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:20.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:20.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:20.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:20.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:20.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:20.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:20.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:20.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:20.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:20 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:02:20.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:20 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:02:20.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:20.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:02:20.674 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:20 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:02:20.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:20 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:02:20.708 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:20 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:02:20.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:20 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:02:20.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:20.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:20.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:20.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:20.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:21.014 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:21.014 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:21.022 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:21.093 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:21.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:21.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:02:21.210 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:21.211 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":301,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:20.420254+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:21.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:21.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:21.428 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:21.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:21.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:21.532 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:21 smithi117 ceph-mon[32835]: pgmap v866: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:21.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:21 smithi117 ceph-mon[32835]: osdmap e301: 12 total, 3 up, 11 in 2024-08-06T18:02:21.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2684232105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:21 smithi086 ceph-mon[29229]: pgmap v866: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:21 smithi086 ceph-mon[29229]: osdmap e301: 12 total, 3 up, 11 in 2024-08-06T18:02:21.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2684232105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:21.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:21.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:21.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:21 smithi107 ceph-mon[32381]: pgmap v866: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:21 smithi107 ceph-mon[32381]: osdmap e301: 12 total, 3 up, 11 in 2024-08-06T18:02:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2684232105' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:21.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:02:21.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:21.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:21.968 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:22.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:22.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:22.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:22.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:22.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:22.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:22.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:22.325 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:22.325 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:22.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:22.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:22.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:22 smithi107 ceph-mon[32381]: osdmap e302: 12 total, 3 up, 11 in 2024-08-06T18:02:22.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:22 smithi086 ceph-mon[29229]: osdmap e302: 12 total, 3 up, 11 in 2024-08-06T18:02:22.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:22.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:22.866 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:22.866 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:22 smithi117 ceph-mon[32835]: osdmap e302: 12 total, 3 up, 11 in 2024-08-06T18:02:22.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:22.999 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:22.999 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:23.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:23.109 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:23.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:23.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:23.408 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:23.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:23.614 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:23 smithi086 ceph-mon[29229]: pgmap v869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:23 smithi117 ceph-mon[32835]: pgmap v869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:23.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:23 smithi107 ceph-mon[32381]: pgmap v869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:23.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:23.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:23.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:23.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:24.047 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:24.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:02:24.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:24.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:24.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:24.261 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:24.261 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:24.277 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:24.277 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:24.278 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:24.292 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:24.295 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:24.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:24.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:24.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:24.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:24.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:24.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:24.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:24.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3097260358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:24.688 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3097260358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3097260358' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:24.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:24.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:24.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:25.016 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:25.017 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:25.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:25.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:25.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:25.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:25.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:25.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:25.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:25.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:25 smithi086 ceph-mon[29229]: pgmap v870: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:25.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:25.670 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:25 smithi117 ceph-mon[32835]: pgmap v870: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:25.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:25 smithi107 ceph-mon[32381]: pgmap v870: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:25.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:25.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:25.865 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:25.865 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:25.866 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:25.870 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:25.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:26.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:26.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:26.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:26.338 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:02:26.446 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:26.446 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:26.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:26.508 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:26.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:02:26.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:02:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:26 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:02:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:26 smithi107 ceph-mon[32381]: osdmap e303: 12 total, 3 up, 11 in 2024-08-06T18:02:26.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:02:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:26 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:02:26.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:26 smithi117 ceph-mon[32835]: osdmap e303: 12 total, 3 up, 11 in 2024-08-06T18:02:26.754 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:26.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:26.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:26.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:02:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:26 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3760905943' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:02:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:26 smithi086 ceph-mon[29229]: osdmap e303: 12 total, 3 up, 11 in 2024-08-06T18:02:26.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:27.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:27.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:27.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:27.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:27.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:27 smithi086 ceph-mon[29229]: pgmap v871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:27.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:27.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:27.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:27 smithi107 ceph-mon[32381]: pgmap v871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:27.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:27.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:27.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:27.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:27 smithi117 ceph-mon[32835]: pgmap v871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:27.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:02:27.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:28.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:28.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:28.294 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:28.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:28.350 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:28.452 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:28.452 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":303,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:26.338768+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:28.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:28.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:28.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:28 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:02:28.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:28 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:02:28.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3464707299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:28.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:28.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:28.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:28 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:02:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:28 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:02:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3464707299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:28 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:02:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:28 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:02:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3464707299' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:28.920 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:28.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:02:28.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:29.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:29.207 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:29.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:29.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:29.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:29.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:29 smithi086 ceph-mon[29229]: pgmap v873: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:29.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:29.678 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:29 smithi117 ceph-mon[32835]: pgmap v873: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:29.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:29 smithi107 ceph-mon[32381]: pgmap v873: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:29.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:02:29.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:29.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:30.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:30.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:30.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:30.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:30.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:30.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:30.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:30.666 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:30.677 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:30.678 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:30.678 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:30.696 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:30.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:30.714 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:30.715 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:30.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:30.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:30.778 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:30.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:31.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:31.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:31.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:31.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:31.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:31.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:31.471 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:31.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:02:31.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:31.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:31.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:31 smithi086 ceph-mon[29229]: pgmap v874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1711324120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:31.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:31.720 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:02:31.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:31.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:31 smithi117 ceph-mon[32835]: pgmap v874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:31.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1711324120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:31.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:31.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:31.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:31 smithi107 ceph-mon[32381]: pgmap v874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1711324120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:32.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:32.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:32.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:32.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:32.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:32.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:32.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:32.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:32.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:32.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:32.588 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:32.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:32.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:32.745 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:32.745 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:32.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:32.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:32.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:32.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:33.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:33.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:33.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:33.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:33.383 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:33.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:33.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:33.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:33 smithi086 ceph-mon[29229]: pgmap v875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:33.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:33.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:33.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:33.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:33 smithi117 ceph-mon[32835]: pgmap v875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:33.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:33.675 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:33 smithi107 ceph-mon[32381]: pgmap v875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:33.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:33.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:33.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:33.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:34.041 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:34.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:02:34.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:34.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:02:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:34.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:34.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:34.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:34.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:34.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:34.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:34.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:34.647 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:34.647 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:34.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:34.794 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:34.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:34.923 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:34.923 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":303,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:26.338768+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:35.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:35.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:35.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:35.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:35.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:35.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:35.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:35.480 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:02:35.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:35.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:35.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:35 smithi086 ceph-mon[29229]: pgmap v876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:35.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:35.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3314644283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:35.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:35.846 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:35 smithi107 ceph-mon[32381]: pgmap v876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:35.846 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:35.846 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3314644283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:35 smithi117 ceph-mon[32835]: pgmap v876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3314644283' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:35.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:36.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:36.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:36.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:36.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:36.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:36.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:36.424 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:36.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:36.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:36.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:36 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:36.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:36.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:36 smithi086 ceph-mon[29229]: osdmap e304: 12 total, 3 up, 11 in 2024-08-06T18:02:36.641 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:36 smithi086 ceph-mon[29229]: osdmap e305: 12 total, 3 up, 11 in 2024-08-06T18:02:36.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:36 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:36 smithi117 ceph-mon[32835]: osdmap e304: 12 total, 3 up, 11 in 2024-08-06T18:02:36.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:36 smithi117 ceph-mon[32835]: osdmap e305: 12 total, 3 up, 11 in 2024-08-06T18:02:36.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:36 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:02:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4068137171' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:36 smithi107 ceph-mon[32381]: osdmap e304: 12 total, 3 up, 11 in 2024-08-06T18:02:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:36 smithi107 ceph-mon[32381]: osdmap e305: 12 total, 3 up, 11 in 2024-08-06T18:02:37.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:37.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:37.167 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:37.167 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:37.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:37.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:37.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:37.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:37.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:37.837 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:37 smithi107 ceph-mon[32381]: pgmap v878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:37.838 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:37.838 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:37.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:37 smithi086 ceph-mon[29229]: pgmap v878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:37 smithi117 ceph-mon[32835]: pgmap v878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:37.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:38.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:38.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:38.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:38.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:38.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:38.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:38.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:38.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:38.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:02:39.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:39.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:39.199 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:39.199 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:39.258 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:39.259 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:39.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:39.364 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:39.364 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:39.364 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:39.365 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:39.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:39.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:39.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:39 smithi117 ceph-mon[32835]: pgmap v880: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:39.605 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1305600141' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:39.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:39 smithi086 ceph-mon[29229]: pgmap v880: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:39.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1305600141' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:39.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:39.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:39 smithi107 ceph-mon[32381]: pgmap v880: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1305600141' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:39.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:40.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:40.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:40.062 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:40.063 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:40.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:40.227 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:40.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:40.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:40.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:40.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:40.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:40.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:40.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:40.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:02:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:02:40.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:02:40.999 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:41.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:02:41.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:41.043 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:41.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:41.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:41.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:41.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:41.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:41 smithi107 ceph-mon[32381]: pgmap v881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:02:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:02:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:02:41.728 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:41.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:41.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:41 smithi086 ceph-mon[29229]: pgmap v881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:41.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:02:41.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:02:41.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:02:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:41 smithi117 ceph-mon[32835]: pgmap v881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:02:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:02:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:02:41.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:42.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:02:42.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:42.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:42.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:42.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:42.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:42.397 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:42.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:42.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:42.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:42.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:43.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:43.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:43.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:43.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:43.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:43.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:43.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:43.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:43.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:43 smithi117 ceph-mon[32835]: pgmap v882: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:43.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:43.716 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:43 smithi086 ceph-mon[29229]: pgmap v882: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:43.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:43 smithi107 ceph-mon[32381]: pgmap v882: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:43.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:44.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:44.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:44.268 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:44.268 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:44.354 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:44.354 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:44.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:44.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:44.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:44.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:44.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:44.961 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:44.961 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:45.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:45.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:45.205 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:45.205 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:45.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:45.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:45.460 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:45.461 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":305,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:36.341276+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:45.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:45.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:45 smithi086 ceph-mon[29229]: pgmap v883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3626056684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:45.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:45.745 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:45 smithi107 ceph-mon[32381]: pgmap v883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:45.745 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3626056684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:45.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:45 smithi117 ceph-mon[32835]: pgmap v883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3626056684' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:45.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:46.094 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:46.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:46.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:46.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:46.510 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:02:46.596 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:46.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:46.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:46.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:46.803 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:02:46.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:46.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:46.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:47.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:47.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:47.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:47.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:47.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:47 smithi117 ceph-mon[32835]: pgmap v884: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:47 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:47 smithi117 ceph-mon[32835]: osdmap e306: 12 total, 3 up, 11 in 2024-08-06T18:02:47.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:47.737 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:47 smithi086 ceph-mon[29229]: pgmap v884: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:47.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:47 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:47.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:47.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:47 smithi086 ceph-mon[29229]: osdmap e306: 12 total, 3 up, 11 in 2024-08-06T18:02:47.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:47.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:47.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:47.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:47.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:47 smithi107 ceph-mon[32381]: pgmap v884: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:47 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2666327498' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:02:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:47 smithi107 ceph-mon[32381]: osdmap e306: 12 total, 3 up, 11 in 2024-08-06T18:02:47.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:48.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:48.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:48.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:48.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:48.337 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:48.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:48.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:48.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:48.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:48.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:48.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:02:49.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:49.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:02:49.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:49.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:02:49.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:49.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:49.531 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:02:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:02:49.531 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:02:49.552 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:02:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:02:49.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:02:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:02:49.554 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:02:49.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:02:49.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:02:49.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:02:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:02:49.575 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:49.576 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:02:49.577 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:02:49.577 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:02:49.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:49 smithi117 ceph-mon[32835]: pgmap v886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:49.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:49.742 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:49 smithi107 ceph-mon[32381]: pgmap v886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:49.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:49.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:49.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:49 smithi086 ceph-mon[29229]: pgmap v886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:49.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:49.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:49.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:50.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:50.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:50.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:50.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:50.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:50.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:50.291 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:50.321 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:02:50.322 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:02:50.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:50.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:50.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4161953217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:50.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.533 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:50.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4161953217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:50.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:50.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4161953217' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:02:50.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:02:50.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:50.941 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:50.941 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:51.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:51.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:51.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:51.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:51.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:51.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:51.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:51.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:51.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:51.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:51.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:51 smithi117 ceph-mon[32835]: pgmap v887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:51.723 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:51 smithi117 ceph-mon[32835]: osdmap e307: 12 total, 3 up, 11 in 2024-08-06T18:02:51.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:51.858 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:51 smithi107 ceph-mon[32381]: pgmap v887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:51.859 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:51 smithi107 ceph-mon[32381]: osdmap e307: 12 total, 3 up, 11 in 2024-08-06T18:02:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:51 smithi086 ceph-mon[29229]: pgmap v887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:51 smithi086 ceph-mon[29229]: osdmap e307: 12 total, 3 up, 11 in 2024-08-06T18:02:51.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:52.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:52.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:52.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:52.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:52.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:02:52.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:52.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:52.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:52.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:52.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:53.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:53.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:53.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:53.319 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:53.319 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:53.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:53.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:53 smithi086 ceph-mon[29229]: pgmap v889: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:53.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:53.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:53 smithi107 ceph-mon[32381]: pgmap v889: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:53.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:53.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:53 smithi117 ceph-mon[32835]: pgmap v889: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:53.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:53.974 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:02:54.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:54.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:54.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:54.394 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:54.394 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:54.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:54.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:54.520 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:54.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:54.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:54.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:54.780 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:02:54.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:54.802 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:02:54.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:02:55.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:55.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:55.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:02:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:02:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:02:55.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:55.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:55.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:55.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:55.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:55.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:55.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:55 smithi117 ceph-mon[32835]: pgmap v890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:55 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:55 smithi117 ceph-mon[32835]: osdmap e308: 12 total, 3 up, 11 in 2024-08-06T18:02:55.698 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:55.698 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:55.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:55 smithi086 ceph-mon[29229]: pgmap v890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:55.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:55 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:55.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:55.699 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:55 smithi086 ceph-mon[29229]: osdmap e308: 12 total, 3 up, 11 in 2024-08-06T18:02:55.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:55.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:55.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:55 smithi107 ceph-mon[32381]: pgmap v890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:55.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:55 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:02:55.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3634137930' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:02:55.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:55 smithi107 ceph-mon[32381]: osdmap e308: 12 total, 3 up, 11 in 2024-08-06T18:02:55.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:56.114 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:02:56.114 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":308,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:02:54.520673+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:02:56.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:56.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:56.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:56.450 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:02:56.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:02:56.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:56.547 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/686785175' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:56.547 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:56 smithi117 ceph-mon[32835]: osdmap e309: 12 total, 3 up, 11 in 2024-08-06T18:02:56.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:56.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/686785175' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:56.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:56 smithi086 ceph-mon[29229]: osdmap e309: 12 total, 3 up, 11 in 2024-08-06T18:02:56.689 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:56.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:56.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/686785175' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:02:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:56 smithi107 ceph-mon[32381]: osdmap e309: 12 total, 3 up, 11 in 2024-08-06T18:02:57.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:57.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:57.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:57.138 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:57.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:57.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:57.333 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:57.333 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:57.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:57.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:02:57.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:57.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:57 smithi086 ceph-mon[29229]: pgmap v892: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:57.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:57.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:57 smithi117 ceph-mon[32835]: pgmap v892: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:57 smithi107 ceph-mon[32381]: pgmap v892: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:57.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:58.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:58.192 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:58.193 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:58.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:02:58.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:02:58.426 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:02:58.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:02:58.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:02:58.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:58.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:02:58.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:02:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:02:58.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:02:59.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:02:59.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:02:59.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:59.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:02:59.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:02:59.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:02:59 smithi117 ceph-mon[32835]: pgmap v894: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:59.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:02:59.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:02:59 smithi086 ceph-mon[29229]: pgmap v894: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:02:59.786 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:02:59.786 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:02:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:02:59.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:02:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:02:59 smithi107 ceph-mon[32381]: pgmap v894: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:00.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:00.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:00.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:00.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:00.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:00.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:03:00.357 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:03:00.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:00.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:00.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:00.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:00.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:01.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:01.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:01.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:03:01.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:01.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:01.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:01.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:01.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:01.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:01.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:01 smithi107 ceph-mon[32381]: pgmap v895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:01 smithi086 ceph-mon[29229]: pgmap v895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:01 smithi117 ceph-mon[32835]: pgmap v895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:01.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:02.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:02.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:02.183 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:02.184 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:02.201 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:02.201 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:02.223 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:02.223 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:02.223 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:02.223 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:02.224 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:02.240 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:02.241 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:02.241 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:02.241 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:02.241 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:02.241 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:02.242 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:02.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:02.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:02.434 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:02.434 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:02.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:02.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:02.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/493160700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:02.723 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/493160700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:02.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/493160700' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:02.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:03.093 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:03.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:03.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:03.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:03.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:03.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:03.285 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:03.286 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:03.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:03.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:03.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:03.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:03.541 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:03:03.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:03.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:03.798 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:03 smithi107 ceph-mon[32381]: pgmap v896: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:03.798 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.798 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.798 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.799 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:03.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:03.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:03.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:03 smithi086 ceph-mon[29229]: pgmap v896: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:03.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:03 smithi117 ceph-mon[32835]: pgmap v896: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:03.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:04.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:04.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:04.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:04.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:04.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:04.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:04.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:04.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:04 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:04.596 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:04.597 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:04 smithi117 ceph-mon[32835]: osdmap e310: 12 total, 3 up, 11 in 2024-08-06T18:03:04.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:04.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:04 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:04 smithi107 ceph-mon[32381]: osdmap e310: 12 total, 3 up, 11 in 2024-08-06T18:03:04.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:04.862 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:04.863 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:04.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:04 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:04.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/779932778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:04.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:04 smithi086 ceph-mon[29229]: osdmap e310: 12 total, 3 up, 11 in 2024-08-06T18:03:04.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:05.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:05.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:05.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:05.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:05.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:05.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:05.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:03:05.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:05.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:05.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:05.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:05.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:05 smithi086 ceph-mon[29229]: pgmap v898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:05 smithi117 ceph-mon[32835]: pgmap v898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:05.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:05 smithi107 ceph-mon[32381]: pgmap v898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:06.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:06.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:06.048 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:06.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:06.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:06.349 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:06.349 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:06.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:06.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:06.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:06.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:06.848 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:06.848 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:06.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:06.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:03:07.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:07.221 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:03:07.221 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:03:07.221 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:03:07.221 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:03:07.221 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:03:07.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:07.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:07.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:07 smithi086 ceph-mon[29229]: pgmap v899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:07 smithi086 ceph-mon[29229]: osdmap e311: 12 total, 3 up, 11 in 2024-08-06T18:03:07.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:07.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:07.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:07.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:07 smithi117 ceph-mon[32835]: pgmap v899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:07.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:07 smithi117 ceph-mon[32835]: osdmap e311: 12 total, 3 up, 11 in 2024-08-06T18:03:07.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:07 smithi107 ceph-mon[32381]: pgmap v899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:07 smithi107 ceph-mon[32381]: osdmap e311: 12 total, 3 up, 11 in 2024-08-06T18:03:07.743 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:07.747 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:07.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:07.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:07.895 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:03:07.895 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":311,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:03:06.348860+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:03:08.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:08.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:08.181 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:08.181 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:08.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:08.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:08.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:08.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:08.478 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2698954705' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:08.479 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:08.479 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:08.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:03:08.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:08.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2698954705' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:08.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:08.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2698954705' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:08.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:08.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:08.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:08.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:09.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:09.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:09.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:09.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:09 smithi107 ceph-mon[32381]: pgmap v901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:09.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:09.524 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:09 smithi117 ceph-mon[32835]: pgmap v901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:09.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:09 smithi086 ceph-mon[29229]: pgmap v901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:09.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:09.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:09.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:10.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:10.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:10.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:10.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:10.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:10.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:10.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:10.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:11.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:11.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:11.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:11.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:11.529 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:11 smithi086 ceph-mon[29229]: pgmap v902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:11 smithi117 ceph-mon[32835]: pgmap v902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:11.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:11.655 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:11.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:11 smithi107 ceph-mon[32381]: pgmap v902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:11.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:11.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:12.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:12.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:12.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:12.364 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:03:12.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:12.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:12.611 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:12.611 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:12.612 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:12 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:03:12.612 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:12 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:03:12.612 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:12.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:12 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:03:12.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:12 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:03:12.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:12 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:03:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:12 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:03:12.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:12.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:12.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:12.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:12.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:12.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:13.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:13.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:13.289 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:13.290 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:13.290 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:13.306 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:13.306 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:13.307 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:13.307 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:13.307 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:13.324 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:13.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:13.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:13.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:13.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:13 smithi117 ceph-mon[32835]: pgmap v903: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:13.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:13 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:13.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:13.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:13 smithi117 ceph-mon[32835]: osdmap e312: 12 total, 3 up, 11 in 2024-08-06T18:03:13.422 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1275962695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:13.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:13 smithi086 ceph-mon[29229]: pgmap v903: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:13 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:13 smithi086 ceph-mon[29229]: osdmap e312: 12 total, 3 up, 11 in 2024-08-06T18:03:13.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1275962695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:13.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:13.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:13 smithi107 ceph-mon[32381]: pgmap v903: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:13.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:13 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:13.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2802011778' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:13.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:13 smithi107 ceph-mon[32381]: osdmap e312: 12 total, 3 up, 11 in 2024-08-06T18:03:13.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1275962695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:13.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:13.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:14.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:14.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:14.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:14.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:14.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.637 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.637 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.637 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:14.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:14.759 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:14.760 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:14.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:15.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:15.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:03:15.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:15.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:15.289 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:03:15.290 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:03:15.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:15.445 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:15 smithi117 ceph-mon[32835]: pgmap v905: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:15.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:15.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:15 smithi086 ceph-mon[29229]: pgmap v905: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:15.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:15 smithi107 ceph-mon[32381]: pgmap v905: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:15.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:15.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:16.045 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:16.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:03:16.070 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:16.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:16.198 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:16.198 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:16.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:16.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:16.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:16.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:16.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:03:16.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:16.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:17.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:17.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:17.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:17.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:17.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:17 smithi086 ceph-mon[29229]: pgmap v906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:17.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:17 smithi086 ceph-mon[29229]: osdmap e313: 12 total, 3 up, 11 in 2024-08-06T18:03:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:17 smithi117 ceph-mon[32835]: pgmap v906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:17.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:17 smithi117 ceph-mon[32835]: osdmap e313: 12 total, 3 up, 11 in 2024-08-06T18:03:17.693 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:17.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:17 smithi107 ceph-mon[32381]: pgmap v906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:17 smithi107 ceph-mon[32381]: osdmap e313: 12 total, 3 up, 11 in 2024-08-06T18:03:17.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:17.937 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:17.938 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:17.938 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:17.938 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:17.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:18.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:18.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:18.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:18.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:18.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:18.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:18.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:18.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:18.874 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:03:18.874 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":313,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:03:16.352153+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:13.096228+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:14.809776+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:03:18.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:18.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:18.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:19.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:19.053 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:19.060 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:19.060 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:19.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:19.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:19.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:19.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:19 smithi107 ceph-mon[32381]: pgmap v908: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:19.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2299942093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:19.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:19.605 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:19.605 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:19.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:19 smithi086 ceph-mon[29229]: pgmap v908: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:19.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2299942093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:19 smithi117 ceph-mon[32835]: pgmap v908: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2299942093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:19.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:19.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:19.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:20.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:20.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:20.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:20.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:20.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:20.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:20.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:20.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:20.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:20.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:20.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:21.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:21.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:21.311 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:21.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:21.313 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:21.313 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:21.313 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:21.317 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:21.317 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:21.317 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:21.318 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:21.318 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:21.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:21.334 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:21.335 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:21.335 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:21.335 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:21.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:21.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:21.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:21.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:21.567 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:21.567 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:21.567 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:21 smithi086 ceph-mon[29229]: pgmap v909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:21.567 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:03:21.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:03:21.568 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1849671005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:21.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:21.611 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:21 smithi117 ceph-mon[32835]: pgmap v909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:21.611 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:03:21.611 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:03:21.611 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1849671005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:21 smithi107 ceph-mon[32381]: pgmap v909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:03:21.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:03:21.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1849671005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:21.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:21.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:21.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:22.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:22.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:22.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:22.254 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:22.254 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:22.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:22.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:22.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:22.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:22.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:22.555 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.555 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.555 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.555 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:22 smithi086 ceph-mon[29229]: osdmap e314: 12 total, 3 up, 11 in 2024-08-06T18:03:22.559 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:22.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:22 smithi117 ceph-mon[32835]: osdmap e314: 12 total, 3 up, 11 in 2024-08-06T18:03:22.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:22.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:22 smithi107 ceph-mon[32381]: osdmap e314: 12 total, 3 up, 11 in 2024-08-06T18:03:22.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:22.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:22.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:22.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:23.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:23.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:23.255 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:23.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:23.378 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:03:23.390 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:23.390 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:23.391 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:23 smithi086 ceph-mon[29229]: pgmap v911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:23.391 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:03:23.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:23.537 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:23 smithi107 ceph-mon[32381]: pgmap v911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:23.538 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:03:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:23 smithi117 ceph-mon[32835]: pgmap v911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:03:23.647 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:23.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:23.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:23.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:23.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:24.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:24.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:24.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:03:24.256 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:24.256 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:24.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:24.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:24.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:24 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:03:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:24 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:03:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:24 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:03:24.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:24 smithi086 ceph-mon[29229]: osdmap e315: 12 total, 3 up, 11 in 2024-08-06T18:03:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:24 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:03:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:24 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:03:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:24 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:24.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:03:24.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:24 smithi117 ceph-mon[32835]: osdmap e315: 12 total, 3 up, 11 in 2024-08-06T18:03:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:24 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:03:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:24 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:03:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:24 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4210389676' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:03:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:24 smithi107 ceph-mon[32381]: osdmap e315: 12 total, 3 up, 11 in 2024-08-06T18:03:24.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:24.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:24.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:25.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:25.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:25.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:25.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:25.381 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:25.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:25.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:25 smithi086 ceph-mon[29229]: pgmap v913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:25.581 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:25.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:25.676 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:25 smithi117 ceph-mon[32835]: pgmap v913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:25.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:25 smithi107 ceph-mon[32381]: pgmap v913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:25.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:25.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:26.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:26.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:26.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:26.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:26.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:26.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:26.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:26.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:03:26.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:26.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:26.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:27.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:27.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:27.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:27.429 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:27 smithi086 ceph-mon[29229]: pgmap v914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:27.450 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:03:27.450 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":315,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:03:23.371001+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:02:19.933717+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:03:27.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:27.565 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:27 smithi117 ceph-mon[32835]: pgmap v914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:27.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:27.700 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:27 smithi107 ceph-mon[32381]: pgmap v914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:27.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:27.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:27.926 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:27.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:27.942 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:27.942 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:27.942 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:27.942 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:27.943 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:27.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:27.960 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:27.960 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:27.961 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:27.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:27.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:28.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:28.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:28.105 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:28.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:28.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:28.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:28.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:28.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:28.513 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:28.513 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:28.513 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1951975714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:28.514 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2890321769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:28.514 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.514 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:28.649 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1951975714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:28.649 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2890321769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:28.649 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.649 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1951975714' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:28.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2890321769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:28.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:28.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:28.776 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:28.777 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:28.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:29.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:29.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:29.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:29.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:29.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:29.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:29.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:29.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:29.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:29.599 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:29 smithi086 ceph-mon[29229]: pgmap v915: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:29.599 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:29 smithi117 ceph-mon[32835]: pgmap v915: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:29 smithi107 ceph-mon[32381]: pgmap v915: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:29.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:29.839 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:03:29.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:29.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:30.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:30.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:30.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:30.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:30.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:30.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:30.571 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:03:30.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:03:30.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:30.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:03:30.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:30.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:31.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:31.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:31.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:31.398 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:31.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:31.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:31.489 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:31 smithi117 ceph-mon[32835]: pgmap v916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:31.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:31.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:31 smithi086 ceph-mon[29229]: pgmap v916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:31.680 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:31.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:31 smithi107 ceph-mon[32381]: pgmap v916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:31.717 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:31.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:31.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:32.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:32.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:32.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:32.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:32.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:32.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:32.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:32.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:32.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:32.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:32.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:33.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:33.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:33.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:33.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:33.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:33.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:33.488 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:33 smithi086 ceph-mon[29229]: pgmap v917: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:33.489 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:33.489 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:33.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:33.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:33 smithi107 ceph-mon[32381]: pgmap v917: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:33.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:33.750 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:33.750 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:33.750 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:33.750 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:33.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:33 smithi117 ceph-mon[32835]: pgmap v917: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:33.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:34.012 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:34.012 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:34.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:34.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:34.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:34.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:34.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:34.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:34.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:34.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:34.894 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:34.894 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:34.894 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:34.906 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:34.907 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:34.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:34.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:34.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:34.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:34.925 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:34.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:34.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:34.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:34.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:34.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:34.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:35.061 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:35.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:35.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:35.409 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:03:35.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:35.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:35.617 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:35 smithi107 ceph-mon[32381]: pgmap v918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:35.617 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:35.617 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1326329554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:35.618 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.618 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:35.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:35 smithi086 ceph-mon[29229]: pgmap v918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:35.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:35.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1326329554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:35.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.750 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.854 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:03:35.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:35 smithi117 ceph-mon[32835]: pgmap v918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1326329554' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:35.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:36.107 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:36.108 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:36.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:36.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:36.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:36.405 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:36.406 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:36.514 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:03:36.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:03:36.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:36.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:36.690 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:36.690 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:36.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:36.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:36 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:36.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:36.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:36 smithi086 ceph-mon[29229]: osdmap e316: 12 total, 3 up, 11 in 2024-08-06T18:03:36.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:36 smithi086 ceph-mon[29229]: osdmap e317: 12 total, 3 up, 11 in 2024-08-06T18:03:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:36 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:36 smithi107 ceph-mon[32381]: osdmap e316: 12 total, 3 up, 11 in 2024-08-06T18:03:36.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:36 smithi107 ceph-mon[32381]: osdmap e317: 12 total, 3 up, 11 in 2024-08-06T18:03:36.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:36.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:36.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:36 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:36.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2229852897' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:36.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:36 smithi117 ceph-mon[32835]: osdmap e316: 12 total, 3 up, 11 in 2024-08-06T18:03:36.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:36 smithi117 ceph-mon[32835]: osdmap e317: 12 total, 3 up, 11 in 2024-08-06T18:03:36.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:37.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:37.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:37.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:37.370 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:37.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:37.405 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:37.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:37.500 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:37 smithi117 ceph-mon[32835]: pgmap v920: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:37.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:37.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:37 smithi086 ceph-mon[29229]: pgmap v920: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:37.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:37 smithi107 ceph-mon[32381]: pgmap v920: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:37.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:37.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:37.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:37.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:38.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:38.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:38.170 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:38.170 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:38.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:38.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:38.446 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:03:38.446 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:03:38.446 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:38 smithi117 ceph-mon[32835]: osdmap e318: 12 total, 3 up, 11 in 2024-08-06T18:03:38.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:38.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:03:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:03:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:38 smithi107 ceph-mon[32381]: osdmap e318: 12 total, 3 up, 11 in 2024-08-06T18:03:38.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:03:38.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:03:38.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:38 smithi086 ceph-mon[29229]: osdmap e318: 12 total, 3 up, 11 in 2024-08-06T18:03:38.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:38.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:03:38.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:39.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:39.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:39.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:39.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:03:39.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:39.510 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:39 smithi086 ceph-mon[29229]: pgmap v923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:39.510 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:39.511 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:39.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:39.704 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:39.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:39 smithi107 ceph-mon[32381]: pgmap v923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:39.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:39.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:39 smithi117 ceph-mon[32835]: pgmap v923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:39.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:39.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:39.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:39.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:39.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:40.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:40.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:03:40.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:40.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:40.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:40.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:40.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:40.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:03:40.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:03:40.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:40.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:41.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:41.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:41.253 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:41.253 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:41.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:41.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:41.519 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:41 smithi086 ceph-mon[29229]: pgmap v924: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:41.519 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:03:41.519 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:41 smithi086 ceph-mon[29229]: osdmap e319: 12 total, 3 up, 11 in 2024-08-06T18:03:41.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:41 smithi107 ceph-mon[32381]: pgmap v924: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:03:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:41 smithi107 ceph-mon[32381]: osdmap e319: 12 total, 3 up, 11 in 2024-08-06T18:03:41.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:41.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:41.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:41.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:41.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:41 smithi117 ceph-mon[32835]: pgmap v924: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:03:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:41 smithi117 ceph-mon[32835]: osdmap e319: 12 total, 3 up, 11 in 2024-08-06T18:03:41.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:42.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:42.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:42.245 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:03:42.246 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:42.246 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:42.246 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":319,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:03:41.358725+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:03:42.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:42.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:42.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:42.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:42.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:42.831 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.832 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.832 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:03:42.832 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:03:42.832 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.832 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3885020957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3885020957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:42.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:42.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:03:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:03:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:03:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3885020957' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:43.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:43.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:43.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:43.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:43.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:43.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:43.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:43 smithi086 ceph-mon[29229]: pgmap v926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:43.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:43 smithi117 ceph-mon[32835]: pgmap v926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:43.931 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:43 smithi107 ceph-mon[32381]: pgmap v926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:43.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:44.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:44.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:44.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:44.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:44.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:44.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:44.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:44.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:44.778 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:44.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:44.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:45.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:45.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:45.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:45.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:45.583 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:45.583 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:45.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:45.611 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:03:45.692 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:45.692 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:45.692 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:45.709 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:45.710 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:45.710 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:45.710 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:45.710 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:45.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:45.727 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:45.727 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:45.727 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:45.729 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:45.729 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:45.729 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:45.729 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:45.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:45.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:45.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:45 smithi086 ceph-mon[29229]: pgmap v927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:45 smithi117 ceph-mon[32835]: pgmap v927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:45.884 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:45 smithi107 ceph-mon[32381]: pgmap v927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:45.884 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:03:45.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:46.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:46.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:46.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:46.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:46.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:46.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:46.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:46.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:46.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: osdmap e320: 12 total, 3 up, 11 in 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1181984341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:46 smithi086 ceph-mon[29229]: osdmap e321: 12 total, 3 up, 11 in 2024-08-06T18:03:46.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: osdmap e320: 12 total, 3 up, 11 in 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1181984341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:46 smithi107 ceph-mon[32381]: osdmap e321: 12 total, 3 up, 11 in 2024-08-06T18:03:47.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:03:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3723681833' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:03:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: osdmap e320: 12 total, 3 up, 11 in 2024-08-06T18:03:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1181984341' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:47.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:47.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:47.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:46 smithi117 ceph-mon[32835]: osdmap e321: 12 total, 3 up, 11 in 2024-08-06T18:03:47.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:47.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:47.226 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:47.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:47.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:47.487 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:47.487 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:47.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:47.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:47.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:47 smithi086 ceph-mon[29229]: pgmap v929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:47.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:47.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:47 smithi107 ceph-mon[32381]: pgmap v929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:47.988 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:03:47.988 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:03:48.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:47 smithi117 ceph-mon[32835]: pgmap v929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:48.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:48.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:48.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:03:48.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:48.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:48.650 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:48.650 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:48.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:03:48.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:48.731 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:48.737 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:48.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:48.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:48.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:48.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:48.994 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:48.994 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:48.994 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:48.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:49.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:49.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:49.254 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:49.255 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:49.255 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:49.255 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:49.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:49.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:03:49.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:49.545 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:49.546 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:49.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:49.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:49.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:49 smithi086 ceph-mon[29229]: pgmap v931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:49.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:49.947 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:49 smithi117 ceph-mon[32835]: pgmap v931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:49 smithi107 ceph-mon[32381]: pgmap v931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:50.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:50.076 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:50.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:50.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:50.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:50.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:50.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:50.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:50.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:51.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:51.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:51.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:51.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:51.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:51 smithi107 ceph-mon[32381]: pgmap v932: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:51.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:51.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:51.779 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:51 smithi086 ceph-mon[29229]: pgmap v932: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:51.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:51.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:52.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:52.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:52.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:51 smithi117 ceph-mon[32835]: pgmap v932: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 318 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:52.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:52.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:52.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:52.549 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:03:52.549 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":321,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:03:46.360497+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:03:52.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:52.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:52.762 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1679913288' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:52.762 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:52.762 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:52.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:53.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:53.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1679913288' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1679913288' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:03:53.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:53.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:53.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:53.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:53.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:53.699 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:53 smithi107 ceph-mon[32381]: pgmap v933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:53.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:53.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:53 smithi086 ceph-mon[29229]: pgmap v933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:53 smithi117 ceph-mon[32835]: pgmap v933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:53.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:54.058 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:03:54.059 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:03:54.068 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:54.068 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:54.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:54.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:54.235 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:54.235 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:54.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:54.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:54.505 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:54.506 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:54.506 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:54.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:54.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:54.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:54 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:03:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:54 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:03:54.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:54.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:54 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:03:54.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:54 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:03:55.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:55.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:54 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:03:55.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:54 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:03:55.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:03:55.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:55.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:55.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:55.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:55.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:55.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:55.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:55.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:55 smithi117 ceph-mon[32835]: pgmap v934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:55.897 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:55 smithi086 ceph-mon[29229]: pgmap v934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:55.898 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:55.898 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:55 smithi107 ceph-mon[32381]: pgmap v934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:55.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:56.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:56.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:56.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:56.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:03:56.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:56.553 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:56.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:03:56.652 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:03:56.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:56.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:56.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:56.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:03:56.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:03:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:03:56.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:03:56.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:57.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:57.073 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:03:57.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:57.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:03:57.251 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:03:57.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:03:57.265 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:03:57.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:03:57.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:03:57.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:03:57.267 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:03:57.284 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:03:57.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:03:57.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:03:57.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:57.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:57.649 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:03:57.649 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:03:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:03:57.727 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:57.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: pgmap v935: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: osdmap e322: 12 total, 3 up, 11 in 2024-08-06T18:03:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2393326053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:57.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:57.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: pgmap v935: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: osdmap e322: 12 total, 3 up, 11 in 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2393326053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:57.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:58.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: pgmap v935: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:58.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:03:58.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/131521727' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:03:58.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: osdmap e322: 12 total, 3 up, 11 in 2024-08-06T18:03:58.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2393326053' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:03:58.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:58.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:58.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:03:58.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:03:58.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:58.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.689 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:03:58.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:03:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:03:59.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:03:59.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:59.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:03:59.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:03:59.482 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:59.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:59.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:03:59.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:03:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:03:59.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:03:59.625 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:03:59.626 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:03:59.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:03:59.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:03:59 smithi086 ceph-mon[29229]: pgmap v937: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:59.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:03:59.888 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:03:59 smithi107 ceph-mon[32381]: pgmap v937: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:03:59.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:03:59 smithi117 ceph-mon[32835]: pgmap v937: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:00.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:00.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:00.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:00.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:00.555 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:00.555 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:00.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:00.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:00.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:01.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:01.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:04:01.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:01.309 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:04:01.309 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:04:01.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:240,from:0,reservation_nonce:0)", 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 342.543212584, 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00019324500000000001, 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736386+0000", 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2099999999999995e-07 2024-08-06T18:04:01.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736390+0000", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3009999999999998e-06 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736400+0000", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.065e-05 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736408+0000", 2024-08-06T18:04:01.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6809999999999998e-06 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736448+0000", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0349999999999998e-05 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736461+0000", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3015e-05 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.736579+0000", 2024-08-06T18:04:01.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000117427 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 342.542500115, 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5890999999999996e-05, 2024-08-06T18:04:01.314 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.315 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737098+0000", 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.53e-07 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737101+0000", 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8830000000000002e-06 2024-08-06T18:04:01.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737109+0000", 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6899999999999992e-06 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737113+0000", 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9489999999999998e-06 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737147+0000", 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4094000000000003e-05 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737154+0000", 2024-08-06T18:04:01.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1760000000000004e-06 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:58:18.737174+0000", 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9746e-05 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:259,from:0,reservation_nonce:0)", 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 260.50792602199999, 2024-08-06T18:04:01.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000226534, 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771673+0000", 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0399999999999995e-07 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771676+0000", 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7480000000000002e-06 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771685+0000", 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1759999999999999e-06 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771690+0000", 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2950000000000001e-06 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771734+0000", 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218000000000003e-05 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771750+0000", 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6150999999999998e-05 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771899+0000", 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014824200000000001 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:04:01.323 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 260.50720764200003, 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8885000000000002e-05, 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:04:01.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0699999999999998e-07 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772394+0000", 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5969999999999999e-06 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772400+0000", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9539999999999998e-06 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772404+0000", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6399999999999999e-06 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772435+0000", 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0793999999999997e-05 2024-08-06T18:04:01.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772441+0000", 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.224e-06 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772459+0000", 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8269000000000001e-05 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.327 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:303,from:0,reservation_nonce:0)", 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 94.241679653000006, 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021705499999999999, 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:04:01.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:04:01.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6899999999999996e-07 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037923+0000", 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2150000000000001e-06 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037932+0000", 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2110000000000004e-06 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037937+0000", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6019999999999999e-06 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037982+0000", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5108000000000002e-05 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038001+0000", 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8410000000000002e-05 2024-08-06T18:04:01.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038136+0000", 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013474 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 94.241011817, 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1237999999999998e-05, 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:04:01.332 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:04:01.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038587+0000", 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9999999999999997e-07 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038591+0000", 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6270000000000002e-06 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038599+0000", 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4489999999999999e-06 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038604+0000", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5190000000000003e-06 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038634+0000", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0531999999999998e-05 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038640+0000", 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5099999999999998e-06 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:04:01.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038658+0000", 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8000999999999999e-05 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:04:01.336 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:04:01.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:01.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:01.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:01.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:04:01.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:01.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:01 smithi107 ceph-mon[32381]: pgmap v938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:01.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:01 smithi107 ceph-mon[32381]: osdmap e323: 12 total, 3 up, 11 in 2024-08-06T18:04:01.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:02.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:02.040 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:02.041 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:02.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:01 smithi086 ceph-mon[29229]: pgmap v938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:02.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:01 smithi086 ceph-mon[29229]: osdmap e323: 12 total, 3 up, 11 in 2024-08-06T18:04:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:01 smithi117 ceph-mon[32835]: pgmap v938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:01 smithi117 ceph-mon[32835]: osdmap e323: 12 total, 3 up, 11 in 2024-08-06T18:04:02.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:04:02.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:02.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:02.322 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:02.322 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:02.383 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:02.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:02.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:02.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:02.712 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:02.750 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:02.829 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:02.829 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":323,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:01.364619+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:02.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:02.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:03.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:03.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:03.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:03.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:03.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:03.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:03.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:03.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:03.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:03.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:03.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:03.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:03.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:03 smithi086 ceph-mon[29229]: pgmap v940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:03.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/579250050' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:03 smithi117 ceph-mon[32835]: pgmap v940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/579250050' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:03.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:03.918 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:03 smithi107 ceph-mon[32381]: pgmap v940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:03.918 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/579250050' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:04.015 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:04.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:04.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:04.182 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:04.183 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:04.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:04.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:04.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:04.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:04.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:04.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:04.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:04.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:04:04.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:04.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:04.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:05.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:05.104 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:05.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:05.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:05.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:05.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:05.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:05.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:05.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:05.811 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:05.812 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:05.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:05 smithi086 ceph-mon[29229]: pgmap v941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:05.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:05.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:05 smithi107 ceph-mon[32381]: pgmap v941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:06.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:05 smithi117 ceph-mon[32835]: pgmap v941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:06.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:06.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:06.364 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:06.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:06.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:06.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:06.634 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:06.635 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:06.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:06.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:06.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:06.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:06.669 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:06.670 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:06.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:06.797 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:06.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:06.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:07.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:07.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:07.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:07.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:07.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:07.379 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:07.379 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:07.421 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:07.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:07.507 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:07 smithi117 ceph-mon[32835]: pgmap v942: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:07.508 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3141682414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:07.508 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.508 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.508 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:07.547 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:07.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:04:07.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:07.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:07 smithi086 ceph-mon[29229]: pgmap v942: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3141682414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:07.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:07.666 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:04:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:07 smithi107 ceph-mon[32381]: pgmap v942: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3141682414' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:07.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:07.799 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:07.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:08.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:08.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:08.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:08.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:08.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:08.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:08.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:08.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:08 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:08.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:08.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:08 smithi107 ceph-mon[32381]: osdmap e324: 12 total, 3 up, 11 in 2024-08-06T18:04:08.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:08.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:08.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:08 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:08.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:08.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:08 smithi117 ceph-mon[32835]: osdmap e324: 12 total, 3 up, 11 in 2024-08-06T18:04:08.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:08.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:08.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:08 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:08.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1456914970' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:08.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:08 smithi086 ceph-mon[29229]: osdmap e324: 12 total, 3 up, 11 in 2024-08-06T18:04:08.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:08.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:09.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:09.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:09.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:09.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:09.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:09.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:04:09.441 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:09.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:09.632 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:09.670 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:09.670 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:09.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:09.882 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:09.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:09.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: pgmap v944: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:09.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3653353179,config=507150784,health=3508906169,kv=2809073277,logm=289972259}) 2024-08-06T18:04:09.929 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470414056}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078396194}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206623822}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084939564}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664262378}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:04:09.930 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114695543}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724971269}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566288176}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631043160}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1715749666}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808746770}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133773441}) 2024-08-06T18:04:09.931 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484559898}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771071947}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336150783}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052456396}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482387354}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3312110071}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806313730}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311865453}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3702642497}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030129009}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3153234020}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21147094}) 2024-08-06T18:04:09.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=18} crc {logm=1678884639,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1315929674,mon_config_key=3949929078}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=566014017}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=165434554}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=21} crc {mon_config_key=3870484500,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=952881640,osdmap=3370532433}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1140188073}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4276649800}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=70395650}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:04:09.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1653213536}) 2024-08-06T18:04:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: pgmap v944: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3653353179,config=507150784,health=3508906169,kv=2809073277,logm=289972259}) 2024-08-06T18:04:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470414056}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078396194}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206623822}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084939564}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664262378}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:04:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114695543}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724971269}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566288176}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631043160}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1715749666}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808746770}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133773441}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484559898}) 2024-08-06T18:04:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771071947}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336150783}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052456396}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482387354}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3312110071}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806313730}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311865453}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3702642497}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030129009}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3153234020}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21147094}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=18} crc {logm=1678884639,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1315929674,mon_config_key=3949929078}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=566014017}) 2024-08-06T18:04:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=165434554}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=21} crc {mon_config_key=3870484500,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=952881640,osdmap=3370532433}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1140188073}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4276649800}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=70395650}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:04:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1653213536}) 2024-08-06T18:04:09.968 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:09.968 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":324,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:07.667294+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:09.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: pgmap v944: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:09.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=30,config=2,health=12,kv=53,logm=3} crc {auth=3653353179,config=507150784,health=3508906169,kv=2809073277,logm=289972259}) 2024-08-06T18:04:09.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=470414056}) 2024-08-06T18:04:09.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1078396194}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2206623822}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1084939564}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1664262378}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:04:09.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4114695543}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=724971269}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566288176}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1631043160}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1715749666}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808746770}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=133773441}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=484559898}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771071947}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=336150783}) 2024-08-06T18:04:09.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1052456396}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3482387354}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3312110071}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1806313730}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3311865453}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3702642497}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030129009}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3153234020}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=21147094}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=41,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=18} crc {logm=1678884639,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1315929674,mon_config_key=3949929078}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=566014017}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=165434554}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=61,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=21} crc {mon_config_key=3870484500,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=952881640,osdmap=3370532433}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1140188073}) 2024-08-06T18:04:09.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4276649800}) 2024-08-06T18:04:09.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=70395650}) 2024-08-06T18:04:09.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:04:09.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=31} crc {osdmap=1653213536}) 2024-08-06T18:04:10.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:10.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:10.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:04:10.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:10.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:10.389 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:10.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:10.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:10.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:10.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/993140022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/993140022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:10.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:10.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/993140022' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:11.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:11.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:11.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:04:11.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:11.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:11.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:11.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:11.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:11.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:11.815 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:11 smithi107 ceph-mon[32381]: pgmap v945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:11.815 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:11 smithi107 ceph-mon[32381]: osdmap e325: 12 total, 3 up, 11 in 2024-08-06T18:04:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:11 smithi117 ceph-mon[32835]: pgmap v945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:11 smithi117 ceph-mon[32835]: osdmap e325: 12 total, 3 up, 11 in 2024-08-06T18:04:11.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:11.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:11 smithi086 ceph-mon[29229]: pgmap v945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:11.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:11 smithi086 ceph-mon[29229]: osdmap e325: 12 total, 3 up, 11 in 2024-08-06T18:04:12.004 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:12.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:12.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:12.283 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:12.284 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:12.284 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:12.302 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:12.303 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:12.303 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:12.303 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:12.303 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:12.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:12.322 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:12.322 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:12.322 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:12.323 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:12.323 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:12.323 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:12.323 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:12.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:12.527 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:12.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:12.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:12.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3278823438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:12.736 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:12.759 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:12.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:04:12.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:12.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3278823438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:12.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:12.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:12.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3278823438' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:12.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:13.007 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:13.093 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:13.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:13.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:13.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:13.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:13.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:13.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:13.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:13.643 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:13.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:13.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:13.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:13 smithi107 ceph-mon[32381]: pgmap v947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:13.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:13.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:13.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:13.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:13.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:13.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:13.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:13 smithi086 ceph-mon[29229]: pgmap v947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:13.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:13.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:14.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:13 smithi117 ceph-mon[32835]: pgmap v947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:14.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:14.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:14.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:14.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:14.452 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:14.452 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:14.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:14.667 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:14.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:14.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:14.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:14.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:15.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:04:15.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:15.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:15.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:04:15.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:15.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:15.555 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:15.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:15.690 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:04:15.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:15.779 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:15 smithi107 ceph-mon[32381]: pgmap v948: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:15.780 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:15.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:15.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:15 smithi086 ceph-mon[29229]: pgmap v948: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:15.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:16.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:15 smithi117 ceph-mon[32835]: pgmap v948: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:16.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:16.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:16.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:16.330 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:16.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:16.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:16.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:16.717 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:16.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:16.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:16 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:16.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:16.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:16 smithi086 ceph-mon[29229]: osdmap e326: 12 total, 3 up, 11 in 2024-08-06T18:04:16.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:16 smithi086 ceph-mon[29229]: osdmap e327: 12 total, 3 up, 11 in 2024-08-06T18:04:16.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:16 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:16.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:16.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:16 smithi107 ceph-mon[32381]: osdmap e326: 12 total, 3 up, 11 in 2024-08-06T18:04:16.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:16 smithi107 ceph-mon[32381]: osdmap e327: 12 total, 3 up, 11 in 2024-08-06T18:04:16.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:16.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:16 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:16.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2420568088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:16.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:16 smithi117 ceph-mon[32835]: osdmap e326: 12 total, 3 up, 11 in 2024-08-06T18:04:16.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:16 smithi117 ceph-mon[32835]: osdmap e327: 12 total, 3 up, 11 in 2024-08-06T18:04:17.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:17.180 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:17.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:17.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:17.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:17.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:17.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:17.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:17.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:17.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:17 smithi086 ceph-mon[29229]: pgmap v950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:17.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:17.908 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:17.909 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:17.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:17 smithi107 ceph-mon[32381]: pgmap v950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:17.913 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:17.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:04:18.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:17 smithi117 ceph-mon[32835]: pgmap v950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:18.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:18.205 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:18.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:18.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:18.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:18.623 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:18.623 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:18.637 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:18.638 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:18.638 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:18.638 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:18.638 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:18.654 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:18.655 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:18.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:18.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:18.656 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:18.656 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:18.656 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:18.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:18.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:04:18.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:18.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/181978816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:18.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:19.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/181978816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:19.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/181978816' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:19.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:19.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:19.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:19.303 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:19.303 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:19.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:19.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:19.562 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:19.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:19.674 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:19.674 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":327,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:16.369352+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:23.166724+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:19.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:19 smithi117 ceph-mon[32835]: pgmap v952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1550864318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:19.876 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:04:19.876 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:04:19.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:19.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:19 smithi107 ceph-mon[32381]: pgmap v952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:19.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1550864318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:19.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:19 smithi086 ceph-mon[29229]: pgmap v952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:19.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:19.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1550864318' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:20.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:20.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:20.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:20.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:20.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:04:20.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:20.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:20.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:20.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:20.789 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:20.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:20.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:21.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:21.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:21.105 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:21.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:21.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:04:21.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:21.377 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:21.377 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:21.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:21.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:21.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:21.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:21 smithi086 ceph-mon[29229]: pgmap v953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:21.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:21 smithi107 ceph-mon[32381]: pgmap v953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:21.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:21.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:22.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:22.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:22.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:21 smithi117 ceph-mon[32835]: pgmap v953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:22.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:22.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:22.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:22.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:22.592 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:04:22.592 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:04:22.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:22.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:22.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:22.722 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:04:22.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:22.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:22.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:22.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:23.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:23.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:23.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:04:23.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:23.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:23.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:23.577 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:23.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:23.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:23.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:23.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:23.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:23 smithi086 ceph-mon[29229]: pgmap v954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:23 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:23 smithi086 ceph-mon[29229]: osdmap e328: 12 total, 3 up, 11 in 2024-08-06T18:04:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:23 smithi107 ceph-mon[32381]: pgmap v954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:23 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:23 smithi107 ceph-mon[32381]: osdmap e328: 12 total, 3 up, 11 in 2024-08-06T18:04:23.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:23.980 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:23.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:23.988 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:23.988 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:23.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:23.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:23.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:23.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:24.007 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:24.008 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:24.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:24.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:24.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:24.009 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:24.009 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:24.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:23 smithi117 ceph-mon[32835]: pgmap v954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:24.019 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:23 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:24.019 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2393086695' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:24.019 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:23 smithi117 ceph-mon[32835]: osdmap e328: 12 total, 3 up, 11 in 2024-08-06T18:04:24.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:24.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:24.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:24.418 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:24.418 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:24.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:24.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:24.763 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:24.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:24.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:24.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/150662336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:24.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:24.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:24.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:25.072 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:25.072 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:25.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/150662336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:25.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/150662336' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:25.139 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:219,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:57:10.175377+0000", 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 434.96508323799998, 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00023952700000000001, 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.140 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175377+0000", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175377+0000", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175379+0000", 2024-08-06T18:04:25.141 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.342e-06 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175383+0000", 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.1189999999999999e-06 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175395+0000", 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.203e-05 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.142 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175402+0000", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2579999999999998e-06 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175477+0000", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5011000000000004e-05 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175493+0000", 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5634e-05 2024-08-06T18:04:25.143 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.175617+0000", 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012413300000000001 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:57:10.176172+0000", 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 434.96428894100001, 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4728000000000005e-05, 2024-08-06T18:04:25.144 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176172+0000", 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176172+0000", 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.145 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176172+0000", 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.8500000000000001e-07 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176175+0000", 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.0070000000000002e-06 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.146 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176183+0000", 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.9519999999999994e-06 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176187+0000", 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.9020000000000001e-06 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176238+0000", 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0642000000000001e-05 2024-08-06T18:04:25.147 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176244+0000", 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6200000000000001e-06 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:57:10.176266+0000", 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2019999999999999e-05 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.148 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:240,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:58:18.286505+0000", 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 366.853955052, 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027755500000000002, 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286505+0000", 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.149 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286505+0000", 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286507+0000", 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5689999999999999e-06 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.150 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286512+0000", 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.6920000000000001e-06 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286520+0000", 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7569999999999993e-06 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286547+0000", 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6449e-05 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.151 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286607+0000", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.0435e-05 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286628+0000", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0914e-05 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.286783+0000", 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00015473900000000001 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.152 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:58:18.287403+0000", 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 366.85305796900002, 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6290000000000001e-05, 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287403+0000", 2024-08-06T18:04:25.153 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287403+0000", 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287403+0000", 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.5700000000000002e-07 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.154 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287406+0000", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.348e-06 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287415+0000", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.3319999999999992e-06 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287431+0000", 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6342999999999999e-05 2024-08-06T18:04:25.155 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287468+0000", 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6381000000000003e-05 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287476+0000", 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8440000000000004e-06 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.156 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:58:18.287499+0000", 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2484999999999999e-05 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:260,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:59:42.145702+0000", 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.994758348, 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000310737, 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.157 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145702+0000", 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145702+0000", 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.158 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145703+0000", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.119e-06 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145708+0000", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.25e-06 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145740+0000", 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2273000000000001e-05 2024-08-06T18:04:25.159 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145761+0000", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0747999999999999e-05 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145825+0000", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4042999999999997e-05 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.145847+0000", 2024-08-06T18:04:25.160 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2282000000000001e-05 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146013+0000", 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00016602199999999999 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T17:59:42.146667+0000", 2024-08-06T18:04:25.161 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 282.99379360500001, 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.3181000000000001e-05, 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146667+0000", 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.162 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146667+0000", 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146668+0000", 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.7199999999999998e-07 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146672+0000", 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.968e-06 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.163 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146684+0000", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.259e-05 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146691+0000", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9099999999999999e-06 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146730+0000", 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8801999999999999e-05 2024-08-06T18:04:25.164 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146739+0000", 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.2059999999999996e-06 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T17:59:42.146760+0000", 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1033e-05 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.165 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:275,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:00:48.082567+0000", 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.057893186, 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00027922, 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082567+0000", 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.166 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082567+0000", 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082571+0000", 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4800000000000001e-06 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.167 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082576+0000", 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7999999999999998e-06 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082589+0000", 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3154e-05 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082598+0000", 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.6169999999999996e-06 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.168 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082689+0000", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.0150999999999998e-05 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082707+0000", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8139e-05 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.082847+0000", 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013987900000000001 2024-08-06T18:04:25.169 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:00:48.083413+0000", 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 217.05704798900001, 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014640700000000001, 2024-08-06T18:04:25.170 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083413+0000", 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.171 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083413+0000", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083413+0000", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.3099999999999997e-07 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083417+0000", 2024-08-06T18:04:25.172 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6200000000000001e-06 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083428+0000", 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1025e-05 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083444+0000", 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6169999999999999e-05 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.173 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083526+0000", 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.2402000000000002e-05 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083535+0000", 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.8140000000000007e-06 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:00:48.083559+0000", 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3745000000000001e-05 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.174 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:293,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 144.91770678200001, 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000202034, 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.175 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222755+0000", 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0979999999999999e-06 2024-08-06T18:04:25.176 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222758+0000", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5659999999999998e-06 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222770+0000", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1504e-05 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222779+0000", 2024-08-06T18:04:25.177 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4970000000000007e-06 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222841+0000", 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2020999999999995e-05 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222854+0000", 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2199e-05 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.178 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222956+0000", 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102149 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 144.91693811600001, 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1451000000000003e-05, 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.179 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.180 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223523+0000", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8799999999999998e-07 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223526+0000", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7580000000000001e-06 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223532+0000", 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5570000000000001e-06 2024-08-06T18:04:25.181 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223537+0000", 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7469999999999996e-06 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223581+0000", 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3791000000000001e-05 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.182 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223587+0000", 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9290000000000003e-06 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223604+0000", 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7280999999999999e-05 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:311,from:0,reservation_nonce:0)", 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:04:25.183 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.910758317000003, 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026581800000000002, 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:04:25.184 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229704+0000", 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.66e-06 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229709+0000", 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8219999999999997e-06 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.185 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229722+0000", 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3616e-05 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229748+0000", 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5868999999999999e-05 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229802+0000", 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4267000000000001e-05 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.186 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229823+0000", 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0241000000000002e-05 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229968+0000", 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014534300000000001 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.187 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.909845532000006, 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105858, 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.188 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230616+0000", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0299999999999998e-07 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230620+0000", 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8360000000000002e-06 2024-08-06T18:04:25.189 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230629+0000", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7070000000000004e-06 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230635+0000", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4620000000000004e-06 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230679+0000", 2024-08-06T18:04:25.190 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3914e-05 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230698+0000", 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9364e-05 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230721+0000", 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2872000000000001e-05 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.191 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.192 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:04:25.192 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:04:25.192 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:04:25.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:25.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:25.197 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:25.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:25.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:25.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:25.607 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:25.607 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:25.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:25.747 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:25 smithi107 ceph-mon[32381]: pgmap v956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:25.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:25.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:25 smithi086 ceph-mon[29229]: pgmap v956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:25 smithi117 ceph-mon[32835]: pgmap v956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:25.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:04:25.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:26.000 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:26.000 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:26.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:26.226 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:26.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:26.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:26.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:26.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:04:26.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:26.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:26.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:26 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:04:26.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:26 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:04:26.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:04:26.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:04:26.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:26 smithi107 ceph-mon[32381]: osdmap e329: 12 total, 3 up, 11 in 2024-08-06T18:04:26.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:27.043 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:27.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:27.075 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:26 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:04:27.075 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:26 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:04:27.075 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:04:27.076 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:04:27.076 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:26 smithi117 ceph-mon[32835]: osdmap e329: 12 total, 3 up, 11 in 2024-08-06T18:04:27.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:26 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:04:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:26 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:04:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:04:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:04:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:26 smithi086 ceph-mon[29229]: osdmap e329: 12 total, 3 up, 11 in 2024-08-06T18:04:27.167 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:27.167 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":329,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:26.372393+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:27.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:27.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:04:27.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:27.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:27.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:27.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:27.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:27.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:27.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:27.862 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:27 smithi107 ceph-mon[32381]: pgmap v957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:27.862 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/573249886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:27.865 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:27.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:28.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:27 smithi086 ceph-mon[29229]: pgmap v957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:28.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/573249886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:28.004 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:28.004 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:28.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:27 smithi117 ceph-mon[32835]: pgmap v957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/573249886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:28.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:28.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:28.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:28.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:04:28.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:28.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:28.862 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:28.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:29.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:29.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:29.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:29.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:29.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:29.463 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:29.463 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:29.463 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:29.505 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:29.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:29.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:29.745 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:04:29.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:29 smithi107 ceph-mon[32381]: pgmap v959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:29.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:30.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:29 smithi117 ceph-mon[32835]: pgmap v959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:30.006 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:30.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:29 smithi086 ceph-mon[29229]: pgmap v959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:30.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:30.067 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:30.067 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:30.068 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:30.068 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:30.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:30.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:30.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:30.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:30.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:30.620 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:30.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:30.642 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:30.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:30.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:30.924 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:30.925 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:30.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:30 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:30.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:30.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:30 smithi107 ceph-mon[32381]: osdmap e330: 12 total, 3 up, 11 in 2024-08-06T18:04:31.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:31.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:30 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:31.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:31.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:30 smithi086 ceph-mon[29229]: osdmap e330: 12 total, 3 up, 11 in 2024-08-06T18:04:31.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:31.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:31.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:31.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:31.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:30 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:31.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1037133886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:31.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:30 smithi117 ceph-mon[32835]: osdmap e330: 12 total, 3 up, 11 in 2024-08-06T18:04:31.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:31.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:31.356 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:31.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:04:31.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:31.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:31.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:31.608 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:04:31.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:31.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:31.848 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:31.849 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:31.849 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:31 smithi107 ceph-mon[32381]: pgmap v961: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:31.849 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:31 smithi107 ceph-mon[32381]: osdmap e331: 12 total, 3 up, 11 in 2024-08-06T18:04:31.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:31.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:31 smithi117 ceph-mon[32835]: pgmap v961: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:31.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:31 smithi117 ceph-mon[32835]: osdmap e331: 12 total, 3 up, 11 in 2024-08-06T18:04:32.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:31 smithi086 ceph-mon[29229]: pgmap v961: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:32.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:31 smithi086 ceph-mon[29229]: osdmap e331: 12 total, 3 up, 11 in 2024-08-06T18:04:32.019 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:32.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:32.037 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:32.037 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:32.038 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:32.054 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:32.056 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:32.057 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:32.057 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:32.057 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:32.057 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:32.057 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:32.058 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:32.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:32.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:32.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:32.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:32.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:32.701 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:32.702 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:32.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:32.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:32.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:04:32.907 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:32.907 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:32.907 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:32.907 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:32.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3604370183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:32.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:32.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:32.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:32.921 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:32.922 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:32.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3604370183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:32.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:32.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:32.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:33.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3604370183' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:33.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:33.173 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:33.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:33.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:33.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:33.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:33.710 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:33.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:04:33.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:33.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:33.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:33 smithi117 ceph-mon[32835]: pgmap v963: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:33.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:33 smithi107 ceph-mon[32381]: pgmap v963: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:34.008 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:34.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:34.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:33 smithi086 ceph-mon[29229]: pgmap v963: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:34.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:34.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:34.438 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:34.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:34.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:34.630 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:34.630 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":331,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:31.373777+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:20.430333+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:34.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:34.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:34.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:04:34.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3995973693' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:34.945 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:34.945 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:34.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:35.021 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3995973693' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:35.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:35.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3995973693' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:35.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:35.280 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:35.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:35.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:35.526 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:35.605 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:35.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:35.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:35.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:35.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:35 smithi117 ceph-mon[32835]: pgmap v964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:35.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:35.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:35 smithi086 ceph-mon[29229]: pgmap v964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:35 smithi107 ceph-mon[32381]: pgmap v964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:36.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:36.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:36.304 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:36.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:04:36.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:36.333 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:36.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:36.479 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:36.479 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:36.556 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:36.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:36.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:36.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:36.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:37.006 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:37.007 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:37.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:37.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:37.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:37.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:37.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:37.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:37.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:37.763 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:37.764 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:37.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:37 smithi117 ceph-mon[32835]: pgmap v965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:37 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:04:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:37 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:04:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:37 smithi117 ceph-mon[32835]: osdmap e332: 12 total, 3 up, 11 in 2024-08-06T18:04:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:37.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:37.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:37.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:37 smithi107 ceph-mon[32381]: pgmap v965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:37 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:04:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:37 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:04:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:37 smithi107 ceph-mon[32381]: osdmap e332: 12 total, 3 up, 11 in 2024-08-06T18:04:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:38.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:38.040 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:04:38.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:38.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:38.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:38.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:38.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:37 smithi086 ceph-mon[29229]: pgmap v965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:38.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:37 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:04:38.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:37 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:04:38.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:37 smithi086 ceph-mon[29229]: osdmap e332: 12 total, 3 up, 11 in 2024-08-06T18:04:38.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:04:38.070 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:38.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:38.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:38.316 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:38.316 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:38.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:38.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:38.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:38.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:38.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:04:38.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:38.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:38 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:38 smithi117 ceph-mon[32835]: osdmap e333: 12 total, 3 up, 11 in 2024-08-06T18:04:38.877 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:38.892 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:38.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:38 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:38 smithi107 ceph-mon[32381]: osdmap e333: 12 total, 3 up, 11 in 2024-08-06T18:04:38.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:38 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:38.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/111151093' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:04:38.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:38 smithi086 ceph-mon[29229]: osdmap e333: 12 total, 3 up, 11 in 2024-08-06T18:04:38.981 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:38.981 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:39.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:39.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:39.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:39.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:39.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:39.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:39.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:39 smithi086 ceph-mon[29229]: pgmap v968: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:39.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/411314634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:39.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:40.045 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:40.045 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:40.046 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:40.047 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:40.047 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:40.047 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:40.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:39 smithi117 ceph-mon[32835]: pgmap v968: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:40.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/411314634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:40.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:40.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:39 smithi107 ceph-mon[32381]: pgmap v968: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:40.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/411314634' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:40.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:40.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:40.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:40.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:40.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:40.499 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:40.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:40.708 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:40.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:40.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:40.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:04:40.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:41.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:41.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:41.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:04:41.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:41.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:41.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:41.472 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:41.510 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:41.511 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:41.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:41.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:41.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:41.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:41 smithi107 ceph-mon[32381]: pgmap v969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:41.992 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:41.992 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":333,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:37.764918+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:42.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:41 smithi086 ceph-mon[29229]: pgmap v969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:42.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:41 smithi117 ceph-mon[32835]: pgmap v969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:42.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:42.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:42.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:42.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:42.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:42.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:42.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:42.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:42.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:04:42.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:42.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4141860384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:42.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4141860384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.009 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.034 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:43.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4141860384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:43.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:43.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:43.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:43.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:43.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:43.784 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:43.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:04:43.788 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:43.788 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:43.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:44.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:44.055 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:43 smithi107 ceph-mon[32381]: pgmap v970: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:44.055 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:04:44.055 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:04:44.055 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:44.090 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:43 smithi086 ceph-mon[29229]: pgmap v970: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:04:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:04:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:43 smithi117 ceph-mon[32835]: pgmap v970: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:04:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:04:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:04:44.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:44.266 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:44.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:44.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:44.438 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:44.439 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:44.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:44.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:44.749 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:44.750 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:44.795 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:04:44.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:44.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:45.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:45.112 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:45.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:45.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:45.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:45.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:45.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:45.616 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:45.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:45.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:45.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:45.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:45.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:46.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:46.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:45 smithi107 ceph-mon[32381]: pgmap v971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:46.045 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:45 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:46.045 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:46.045 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:45 smithi107 ceph-mon[32381]: osdmap e334: 12 total, 3 up, 11 in 2024-08-06T18:04:46.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:45 smithi086 ceph-mon[29229]: pgmap v971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:46.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:45 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:46.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:46.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:45 smithi086 ceph-mon[29229]: osdmap e334: 12 total, 3 up, 11 in 2024-08-06T18:04:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:45 smithi117 ceph-mon[32835]: pgmap v971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:45 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1501824898' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:45 smithi117 ceph-mon[32835]: osdmap e334: 12 total, 3 up, 11 in 2024-08-06T18:04:46.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:46.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:04:46.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:46.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:46.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:46.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:46.510 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:46.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:46.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:46.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:46.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:46.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:46.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:46.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:46.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:46.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:46.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:46.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:46.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:46.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:47.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:47.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:47.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:47.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:47.378 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:47.378 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:47.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:47.478 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:47.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:47.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:47 smithi086 ceph-mon[29229]: pgmap v973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:47.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:47 smithi086 ceph-mon[29229]: osdmap e335: 12 total, 3 up, 11 in 2024-08-06T18:04:47.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/309397898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:47.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:47 smithi117 ceph-mon[32835]: pgmap v973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:47.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:47 smithi117 ceph-mon[32835]: osdmap e335: 12 total, 3 up, 11 in 2024-08-06T18:04:47.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/309397898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:47.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:47.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:04:47.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:47 smithi107 ceph-mon[32381]: pgmap v973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:47 smithi107 ceph-mon[32381]: osdmap e335: 12 total, 3 up, 11 in 2024-08-06T18:04:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/309397898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:47.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:47.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:47.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:47.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:48.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:48.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:48.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:48.120 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:48.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:48.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:48.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:48.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:48.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:48.614 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:48.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:48.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:48.773 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:48.773 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:48.805 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:48.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:48.833 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:48.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:04:48.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:49.079 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:49.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:49.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:49.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:49.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:04:49.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:49.528 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:49 smithi117 ceph-mon[32835]: pgmap v975: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:49.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:49.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:49 smithi086 ceph-mon[29229]: pgmap v975: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:49 smithi107 ceph-mon[32381]: pgmap v975: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:49.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:49.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:50.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:50.026 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:50.026 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:50.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:50.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:50.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:50.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:50.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:50.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:50.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:50.514 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:50.514 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":335,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:46.377041+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:03:37.182910+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:50.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:50.619 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:50.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:50.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:50.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:50.996 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:50.996 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:51.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:51.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:51.258 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:51.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:04:51.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:51.396 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:04:51.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:51.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:51.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:51 smithi107 ceph-mon[32381]: pgmap v976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:51.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4294025607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:51.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:51.518 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:51.529 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:51 smithi117 ceph-mon[32835]: pgmap v976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:51.529 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4294025607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:51.529 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:51.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:51.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:51 smithi086 ceph-mon[29229]: pgmap v976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:51.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4294025607' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:04:51.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:04:51.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:51.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:51.839 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:51.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:52.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:52.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:52.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:52.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:52.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:52.645 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:52 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:52.645 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:52.645 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:52 smithi117 ceph-mon[32835]: osdmap e336: 12 total, 3 up, 11 in 2024-08-06T18:04:52.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:52 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:52.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:52.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:52 smithi107 ceph-mon[32381]: osdmap e336: 12 total, 3 up, 11 in 2024-08-06T18:04:52.744 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:52.745 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:52.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:52 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:04:52.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3217317307' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:04:52.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:52 smithi086 ceph-mon[29229]: osdmap e336: 12 total, 3 up, 11 in 2024-08-06T18:04:52.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:52.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:52.918 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:52.918 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:53.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:53.114 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:53.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:53.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:53.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:53.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:53.586 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:53 smithi107 ceph-mon[32381]: pgmap v978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:53.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:53.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:53.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:53 smithi086 ceph-mon[29229]: pgmap v978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:53.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:53.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:53.818 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:53 smithi117 ceph-mon[32835]: pgmap v978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:53.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:53.846 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:53.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:04:53.857 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:53.857 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:53.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:54.093 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:54.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:54.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:54.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:54.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:54.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:54.401 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:04:54.402+0000 7f0c767fc640 0 --2- 172.21.15.86:0/3453382534 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f0c781072b0 0x7f0c78197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:04:54.424 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:04:54.424 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:04:54.424 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:04:54.442 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:04:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:04:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:04:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:04:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:04:54.460 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:04:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:04:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:04:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:04:54.461 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:04:54.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:04:54.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:04:54.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:04:54.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:54.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:54.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:04:54.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:54.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:54.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:54.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:55.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:55.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:04:55.130 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:04:55.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:55.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:55.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:55.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:55.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:55.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:55.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:55.587 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:55.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:04:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: pgmap v979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: osdmap e337: 12 total, 3 up, 11 in 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3453382534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:55.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: pgmap v979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:55.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:04:55.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:04:55.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: osdmap e337: 12 total, 3 up, 11 in 2024-08-06T18:04:55.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3453382534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:55.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: pgmap v979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: osdmap e337: 12 total, 3 up, 11 in 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3453382534' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:04:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:04:55.888 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:55.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:56.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:56.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:56.239 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:56.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:04:56.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:56.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:56.482 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:56.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:56.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:56.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:56.897 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:56.897 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:57.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:04:57.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:04:57.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:04:57.151 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:57.151 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:57.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:57.408 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:04:57.408 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:04:57.408 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:04:57.408 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:04:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:04:57.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:57.413 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:04:57.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:57.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:57 smithi117 ceph-mon[32835]: pgmap v981: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:57.626 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:57.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:57.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:57 smithi107 ceph-mon[32381]: pgmap v981: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:57.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:57.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:57 smithi086 ceph-mon[29229]: pgmap v981: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:57.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:04:57.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:57.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:57.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:57.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:57.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:58.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:58.099 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:04:58.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:04:58.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:04:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:58.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:04:58.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:58 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:58.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:58 smithi117 ceph-mon[32835]: osdmap e338: 12 total, 3 up, 11 in 2024-08-06T18:04:58.626 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:58 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:58.626 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:58.626 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:58 smithi107 ceph-mon[32381]: osdmap e338: 12 total, 3 up, 11 in 2024-08-06T18:04:58.672 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:04:58.672 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:04:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:04:58.673 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:58 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:04:58.673 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2911985277' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:04:58.673 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:58 smithi086 ceph-mon[29229]: osdmap e338: 12 total, 3 up, 11 in 2024-08-06T18:04:58.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:04:58.781 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:04:58.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:04:58.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:58.991 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:04:58.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:59.067 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:04:59.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:04:59.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:04:59.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:59.471 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:04:59 smithi086 ceph-mon[29229]: pgmap v983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:59.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:04:59.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:04:59.681 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:04:59 smithi117 ceph-mon[32835]: pgmap v983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:59.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:04:59 smithi107 ceph-mon[32381]: pgmap v983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:04:59.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:04:59.880 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:04:59.880 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":338,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:04:57.408908+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:04:59.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:00.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:00.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:00.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:05:00.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:00.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:00.511 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/143054432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:00.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:00.596 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:00.619 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/143054432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:00.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:00.620 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:00.620 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:00.620 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:00.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/143054432' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:00.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:00.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:00.885 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:00.886 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:00.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:01.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:01.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:01.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:01.221 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:01.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:05:01.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:01.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:01.472 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:05:01.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:01.582 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:01 smithi117 ceph-mon[32835]: pgmap v984: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:01.582 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:01 smithi117 ceph-mon[32835]: osdmap e339: 12 total, 3 up, 11 in 2024-08-06T18:05:01.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:01.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:01 smithi086 ceph-mon[29229]: pgmap v984: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:01.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:01 smithi086 ceph-mon[29229]: osdmap e339: 12 total, 3 up, 11 in 2024-08-06T18:05:01.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:01.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:01.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:01.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:01.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:01 smithi107 ceph-mon[32381]: pgmap v984: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:01 smithi107 ceph-mon[32381]: osdmap e339: 12 total, 3 up, 11 in 2024-08-06T18:05:01.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:01.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:02.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:02.076 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:02.077 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:02.095 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:02.096 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:02.096 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:02.096 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:02.096 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:02.114 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:02.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:02.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:02.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:02.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:02.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:02.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:02.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:02.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2501598197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:02.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:02.621 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2501598197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:02.622 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:02.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:02.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2501598197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:02.738 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:02.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:02.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:02.903 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:02.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:03.025 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:03.025 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:03.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:03.119 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:03.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:03.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:03.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:03.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:03 smithi117 ceph-mon[32835]: pgmap v986: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:03.543 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.543 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:03 smithi086 ceph-mon[29229]: pgmap v986: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:03 smithi107 ceph-mon[32381]: pgmap v986: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:03.755 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:03.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:05:03.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:03.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:03.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:04.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:04.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:04.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:04.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:04.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:04.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:04.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:04.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:04.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:04.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:04.740 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:04.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:04.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:05:04.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:05.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:05.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:05.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:05.439 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:05:05.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:05.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:05.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:05 smithi107 ceph-mon[32381]: pgmap v987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:05.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:05.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:05.802 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:05 smithi117 ceph-mon[32835]: pgmap v987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:05.802 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:05.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:05 smithi086 ceph-mon[29229]: pgmap v987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:05.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:05.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:06.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:06.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:06.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:06.354 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:06.354 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:06.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:06.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:06 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:06 smithi086 ceph-mon[29229]: osdmap e340: 12 total, 3 up, 11 in 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:06 smithi086 ceph-mon[29229]: osdmap e341: 12 total, 3 up, 11 in 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:06 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:06.616 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:06 smithi117 ceph-mon[32835]: osdmap e340: 12 total, 3 up, 11 in 2024-08-06T18:05:06.617 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:06 smithi117 ceph-mon[32835]: osdmap e341: 12 total, 3 up, 11 in 2024-08-06T18:05:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:06 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1738101832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:06 smithi107 ceph-mon[32381]: osdmap e340: 12 total, 3 up, 11 in 2024-08-06T18:05:06.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:06 smithi107 ceph-mon[32381]: osdmap e341: 12 total, 3 up, 11 in 2024-08-06T18:05:06.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:06.765 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:06.776 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:06.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:06.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:07.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:07.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:07.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:07.424 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:07.424 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:07.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:07.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:07.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:07 smithi086 ceph-mon[29229]: pgmap v989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:07.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:07 smithi107 ceph-mon[32381]: pgmap v989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:07.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:07.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:07 smithi117 ceph-mon[32835]: pgmap v989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:07.909 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:05:07.909 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":341,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:05:06.381936+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:05:07.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:07.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:07.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:08.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:08.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:05:08.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:08.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:08.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:08.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:08.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:08.502 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4257513709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:08.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4257513709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:08.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4257513709' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:08.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:08.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:09.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:09.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:09.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:09.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:09.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:09.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:09.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:09.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:09 smithi086 ceph-mon[29229]: pgmap v991: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:09 smithi117 ceph-mon[32835]: pgmap v991: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:09.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:09.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:09 smithi107 ceph-mon[32381]: pgmap v991: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:09.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:09.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:09.983 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:09.983 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:09.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:09.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:09.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:10.000 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:10.001 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:10.001 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:10.001 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:10.001 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:10.018 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:10.019 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:10.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:10.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:10.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:10.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2857693736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:10.646 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:10.646 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:10.646 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2857693736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:10.646 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.646 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2857693736' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:10.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:10.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:10.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:11.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:11.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:11.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:11.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:11.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:11.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:11 smithi086 ceph-mon[29229]: pgmap v992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:11.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:11.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:11.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:11.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:11.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:11.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:11 smithi107 ceph-mon[32381]: pgmap v992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:11.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:11.739 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:11.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:11.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:11 smithi117 ceph-mon[32835]: pgmap v992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:11.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:11.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:11.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:11.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:12.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:12.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:12.215 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:12.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:12.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:12.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:12.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:12.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:12.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:12.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:12.769 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:12.769 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:12.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:12.931 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:12.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:05:13.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:13.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:13.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:13.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:13.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:13.440 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:13.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:13.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:13 smithi117 ceph-mon[32835]: pgmap v993: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:13.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:13.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:05:13.702 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:13 smithi086 ceph-mon[29229]: pgmap v993: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:13.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:13 smithi107 ceph-mon[32381]: pgmap v993: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:13.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:13.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:14.048 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:14.048 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:14.048 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:14.049 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:14.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:14.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:14.212 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:14.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:14.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:14.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:14.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:14.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:14.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:14.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:14.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:14.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:15.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:15.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:15.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:15.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:15.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:15.368 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:15.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:15.469 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:05:15.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:15.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:15 smithi107 ceph-mon[32381]: pgmap v994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:15.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:15 smithi086 ceph-mon[29229]: pgmap v994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:15.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:15.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:15 smithi117 ceph-mon[32835]: pgmap v994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:15.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:16.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:16.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:16.261 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:16.261 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:16.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:16.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:16.512 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:16.513 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:16.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:16 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:16.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:16.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:16 smithi086 ceph-mon[29229]: osdmap e342: 12 total, 3 up, 11 in 2024-08-06T18:05:16.513 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:16 smithi086 ceph-mon[29229]: osdmap e343: 12 total, 3 up, 11 in 2024-08-06T18:05:16.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:16 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:16 smithi117 ceph-mon[32835]: osdmap e342: 12 total, 3 up, 11 in 2024-08-06T18:05:16.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:16 smithi117 ceph-mon[32835]: osdmap e343: 12 total, 3 up, 11 in 2024-08-06T18:05:16.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:16.781 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:16.781 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:16.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:16 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:16.782 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2664543619' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:16.782 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:16 smithi107 ceph-mon[32381]: osdmap e342: 12 total, 3 up, 11 in 2024-08-06T18:05:16.782 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:16 smithi107 ceph-mon[32381]: osdmap e343: 12 total, 3 up, 11 in 2024-08-06T18:05:16.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:16.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:16.870 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:16.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:17.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:17.045 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:17.045 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:17.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:17.185 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:17.185 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:17.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:17.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:17.473 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:17.474 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:17.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:17.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:17.717 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:17 smithi117 ceph-mon[32835]: pgmap v996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:17.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:17 smithi086 ceph-mon[29229]: pgmap v996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:17.731 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:17.732 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:17.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:17.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:17 smithi107 ceph-mon[32381]: pgmap v996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:18.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:18.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:18.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:18.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:18.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:18.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:18.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:18.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:18.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:05:18.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:18.921 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:18.922 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:18.922 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:18.922 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:19.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:19.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:19.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:19.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:19.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:19.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:19.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:19.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:19 smithi117 ceph-mon[32835]: pgmap v998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:19.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:19.733 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:19 smithi107 ceph-mon[32381]: pgmap v998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:19.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:19 smithi086 ceph-mon[29229]: pgmap v998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:19.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:19.900 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:19.900 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:19.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:19.923 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:19.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:19.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:19.955 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:19.956 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:19.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:20.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:20.238 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:20.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:20.272 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:20.272 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:20.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:20.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:20.523 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:05:20.523 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":343,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:05:16.384820+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:05:20.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/663675012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2218607362' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:20.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:20.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/663675012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:20.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2218607362' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:20.817 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:20.817 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/663675012' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2218607362' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:20.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:21.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:21.092 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:21.093 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:21.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:21.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:21.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:21.485 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:21.485 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:21.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:21 smithi117 ceph-mon[32835]: pgmap v999: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:21.708 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:21.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:05:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:21 smithi107 ceph-mon[32381]: pgmap v999: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:21.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:21.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:21 smithi086 ceph-mon[29229]: pgmap v999: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:21.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:22.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:22.028 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:22.028 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:22.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:22.249 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:22.250 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:22.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:22.279 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:22.343 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:22.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:22.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:22.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:22.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:22.542 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:22.542 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:22.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:22.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:22.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:22.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:22.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:23.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:23.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:23.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:23.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:23.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:23.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:23.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:23.630 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:23.640 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:23 smithi107 ceph-mon[32381]: pgmap v1000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:23.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:23.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:23 smithi086 ceph-mon[29229]: pgmap v1000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:23 smithi117 ceph-mon[32835]: pgmap v1000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:23.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:24.009 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:24.009 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:24.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:24.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:24.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:24.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:24.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:05:24.301 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:24.301 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:24.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:24.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:24.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:24.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:24.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:25.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:25.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:25.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:25.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:25.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:25.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:25 smithi107 ceph-mon[32381]: pgmap v1001: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:25.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:25.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:25 smithi086 ceph-mon[29229]: pgmap v1001: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:25.774 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:25 smithi117 ceph-mon[32835]: pgmap v1001: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:25.870 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:25.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:26.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:26.092 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:26.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:26.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:26.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:26.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:26.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:26.498 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:05:26.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:05:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:26.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:05:26.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:26.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:26.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:05:26.854 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:26.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:05:27.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:27.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:27.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:27.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:27.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:27.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:27 smithi086 ceph-mon[29229]: pgmap v1002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:27 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:05:27.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:27 smithi086 ceph-mon[29229]: osdmap e344: 12 total, 3 up, 11 in 2024-08-06T18:05:27.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:27.657 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:27 smithi117 ceph-mon[32835]: pgmap v1002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:27.657 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:27 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:27.657 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:05:27.657 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:27 smithi117 ceph-mon[32835]: osdmap e344: 12 total, 3 up, 11 in 2024-08-06T18:05:27.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:27.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:27 smithi107 ceph-mon[32381]: pgmap v1002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:27 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/940660818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:05:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:27 smithi107 ceph-mon[32381]: osdmap e344: 12 total, 3 up, 11 in 2024-08-06T18:05:28.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:28.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:28.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:28.235 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:28.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:28.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:28.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:28.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:28.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:28.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:29.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:29.130 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:29.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:29.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:05:29.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:29.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:29.528 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:29 smithi086 ceph-mon[29229]: pgmap v1004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:29.528 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:29.528 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:29.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:29.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:29.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:29 smithi117 ceph-mon[32835]: pgmap v1004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:29.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:29 smithi107 ceph-mon[32381]: pgmap v1004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:30.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:30.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:30.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:30.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:30.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:30.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:30 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:05:30.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:30 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:05:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:30 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:05:30.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:30 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:05:30.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:30.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:30.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:30 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:05:30.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:30 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:05:30.874 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:30.874 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:30.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:31.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:31.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:31.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:31.503 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:05:31.503 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":345,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:05:31.388289+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:36.197619+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:05:31.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:31.541 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:31 smithi117 ceph-mon[32835]: pgmap v1005: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:31.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1516352297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:31.542 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:31 smithi117 ceph-mon[32835]: osdmap e345: 12 total, 3 up, 11 in 2024-08-06T18:05:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:31 smithi086 ceph-mon[29229]: pgmap v1005: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1516352297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:31.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:31 smithi086 ceph-mon[29229]: osdmap e345: 12 total, 3 up, 11 in 2024-08-06T18:05:31.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:31.676 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:31 smithi107 ceph-mon[32381]: pgmap v1005: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:31.676 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1516352297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:31.676 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:31 smithi107 ceph-mon[32381]: osdmap e345: 12 total, 3 up, 11 in 2024-08-06T18:05:31.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:31.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:31.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:31.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:32.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:32.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:32.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:32.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:32.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2694777748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:32.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:32.742 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.742 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.743 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.743 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2694777748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:32.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:32.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2694777748' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:32.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:32.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:32.995 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:32.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:33.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:33.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:33.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:33.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:33.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:33.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:33.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:33.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:33.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:33.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:33 smithi086 ceph-mon[29229]: pgmap v1007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:33.641 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:33.642 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:33.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:33.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:33 smithi117 ceph-mon[32835]: pgmap v1007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:33.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:33.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:33 smithi107 ceph-mon[32381]: pgmap v1007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:33.969 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:33.969 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:34.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:34.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:34.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:34.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:34.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:34.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:34.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:34.620 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:05:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:05:34.620 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:05:34.620 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:05:34.720 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:34.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:34.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:35.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:35.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:35.349 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:35.350 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:35.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:05:35.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:35.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:35.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:35.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:35 smithi086 ceph-mon[29229]: pgmap v1008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:35.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:05:35.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:35.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:35.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:35 smithi117 ceph-mon[32835]: pgmap v1008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:35.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:35 smithi107 ceph-mon[32381]: pgmap v1008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:36.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:36.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:36.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:36.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:36.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:36.509 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:36.510 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:36.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:36.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:36.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:37.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:37.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:37.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:37.302 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:37.302 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:37.303 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:37.303 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:37.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:37.520 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:05:37.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:37.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:37 smithi086 ceph-mon[29229]: pgmap v1009: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:37.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:37.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:37.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:37.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:37.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:37.837 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:37 smithi117 ceph-mon[32835]: pgmap v1009: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:37.837 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:37.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:37 smithi107 ceph-mon[32381]: pgmap v1009: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:38.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:38.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:38.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:38.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:38.517 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:38.517 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:38.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:38.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:38.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:38 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:05:38.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:38.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:38 smithi086 ceph-mon[29229]: osdmap e346: 12 total, 3 up, 11 in 2024-08-06T18:05:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:38 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:05:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:38 smithi117 ceph-mon[32835]: osdmap e346: 12 total, 3 up, 11 in 2024-08-06T18:05:38.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:38.907 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:38 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:05:38.907 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1662239491' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:05:38.908 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:38 smithi107 ceph-mon[32381]: osdmap e346: 12 total, 3 up, 11 in 2024-08-06T18:05:39.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:39.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:39.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:39.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:39.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:39.575 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:39 smithi117 ceph-mon[32835]: pgmap v1011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:39.661 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:39.661 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:39.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:39 smithi086 ceph-mon[29229]: pgmap v1011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:39.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:39.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:39.842 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:39 smithi107 ceph-mon[32381]: pgmap v1011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:39.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:39.974 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:39.974 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:40.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:40.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:40.248 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:40.248 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:40.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:40.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:40.603 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:05:40.603 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:05:40.603 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:05:40.603 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:05:40.603 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:05:40.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:40.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:40.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:40 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:05:40.783 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:40 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:05:40.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:05:40.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:40 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:05:40.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:40 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:05:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:40 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:05:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:40 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:05:40.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:41.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:41.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:41.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:41.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:41.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:41.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:41.531 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:41 smithi086 ceph-mon[29229]: pgmap v1012: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:41.532 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:41.532 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:41.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:41.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:41.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:41.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:41 smithi117 ceph-mon[32835]: pgmap v1012: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:41.856 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:41 smithi117 ceph-mon[32835]: osdmap e347: 12 total, 3 up, 11 in 2024-08-06T18:05:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:41 smithi086 ceph-mon[29229]: osdmap e347: 12 total, 3 up, 11 in 2024-08-06T18:05:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:41 smithi107 ceph-mon[32381]: pgmap v1012: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:41 smithi107 ceph-mon[32381]: osdmap e347: 12 total, 3 up, 11 in 2024-08-06T18:05:41.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:42.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:42.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:42.262 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:42.262 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:42.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:42.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:05:42.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:42.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:42.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:42.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:42.747 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:42.747 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:42.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:42.816 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:05:42.816 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":347,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:05:40.527533+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:26.234999+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:05:42.849 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:42.850 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:42.850 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:42.869 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:42.869 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:42.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:42.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:42.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:42.886 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:42.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:42.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:42.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:42.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:42.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:42.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:42.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:43.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:43.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:43.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:43.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:43.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:43.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:43.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:43.736 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:43.737 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: pgmap v1014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/442581476' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3756333115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:05:43.772 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:43.777 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:43.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:43.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: pgmap v1014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:43.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/442581476' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:43.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3756333115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:43.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: pgmap v1014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/442581476' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3756333115' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:05:43.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:43.998 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:43.998 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:44.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:44.165 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:44.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:44.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:44.458 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:44.459 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:44.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:44.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:44.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:44.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:44.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:44.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:44.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:05:44.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:05:44.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:05:44.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:05:44.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:05:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:05:44.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:05:44.919 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:05:45.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:45.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:45.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:45.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:45.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:45.648 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:45 smithi086 ceph-mon[29229]: pgmap v1015: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:45.649 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:45.649 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:45.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:45.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:45 smithi117 ceph-mon[32835]: pgmap v1015: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:45.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:45 smithi107 ceph-mon[32381]: pgmap v1015: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:46.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:46.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:05:46.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:46.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:46.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:46.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:46.612 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:05:46.612 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:05:46.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:46.656 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:46.656 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:46.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:46.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:46.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:46.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:47.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:47.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:47.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:47.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:05:47.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:47.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:47.590 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:47 smithi107 ceph-mon[32381]: pgmap v1016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:47.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:47 smithi086 ceph-mon[29229]: pgmap v1016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:47.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:47.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:47.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:47 smithi117 ceph-mon[32835]: pgmap v1016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:47.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:48.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:48.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:48.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:48.461 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:48.461 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:48.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:48.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:48.528 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:48.528 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:48.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:48.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:48.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:49.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:49.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:49.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:49.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:49.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:49.549 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:05:49.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:49 smithi117 ceph-mon[32835]: pgmap v1017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:49.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:49.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:49.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:49 smithi086 ceph-mon[29229]: pgmap v1017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:49.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:49 smithi107 ceph-mon[32381]: pgmap v1017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:05:49.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:50.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:50.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:50.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:50.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:50.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:50.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:50.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:50.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:50.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:50.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:50 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:05:50.836 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:50 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:05:50.836 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:50 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:50.836 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:50.836 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:50 smithi117 ceph-mon[32835]: osdmap e348: 12 total, 3 up, 11 in 2024-08-06T18:05:50.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:50.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:50 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:05:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:50 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:05:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:50 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:50 smithi086 ceph-mon[29229]: osdmap e348: 12 total, 3 up, 11 in 2024-08-06T18:05:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:50.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:50 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:05:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:50 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:05:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:50 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:05:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2795386604' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:05:50.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:50 smithi107 ceph-mon[32381]: osdmap e348: 12 total, 3 up, 11 in 2024-08-06T18:05:50.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:51.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:51.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:51.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:51.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:51.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:51.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:51 smithi117 ceph-mon[32835]: pgmap v1019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:51.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:51 smithi117 ceph-mon[32835]: osdmap e349: 12 total, 3 up, 11 in 2024-08-06T18:05:51.750 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:51 smithi086 ceph-mon[29229]: pgmap v1019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:51.750 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:51 smithi086 ceph-mon[29229]: osdmap e349: 12 total, 3 up, 11 in 2024-08-06T18:05:51.751 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:51.751 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:51.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:51.766 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:05:51.766 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:05:51.766 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:05:51.783 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:05:51.784 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:05:51.784 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:05:51.784 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:05:51.784 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:05:51.801 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:05:51.802 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:05:51.802 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:05:51.802 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:05:51.802 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:05:51.802 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:05:51.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:05:51.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:51.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:51 smithi107 ceph-mon[32381]: pgmap v1019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:51.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:51 smithi107 ceph-mon[32381]: osdmap e349: 12 total, 3 up, 11 in 2024-08-06T18:05:52.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:52.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:52.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:52.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:52.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:52.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/964275122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:52.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/964275122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:52.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:52.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/964275122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:05:52.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:52.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:05:52.963 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:05:53.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:53.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:53.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:53.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:53.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:53.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:05:53.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:53.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:53.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:53 smithi117 ceph-mon[32835]: pgmap v1021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:53.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:53.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:53.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:53 smithi086 ceph-mon[29229]: pgmap v1021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:53.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:05:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:53 smithi107 ceph-mon[32381]: pgmap v1021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:53.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:53.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:54.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:54.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:54.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:54.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:54.501 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:05:54.501 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":349,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:05:51.393120+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:05:54.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:54.583 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/64354254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:54.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:54.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:54.847 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/64354254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:54.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:54.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:54.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/64354254' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:05:54.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:05:54.966 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:05:55.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:55.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:55.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:55.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:55.564 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:55.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:55.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:55.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:55.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:55.694 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:05:55.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:05:55.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:55 smithi086 ceph-mon[29229]: pgmap v1022: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:55 smithi117 ceph-mon[32835]: pgmap v1022: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:55.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:55.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:55 smithi107 ceph-mon[32381]: pgmap v1022: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:56.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:56.065 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:56.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:56.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:56.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:56.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:56.306 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:56.306 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:56.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:56.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:56.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:56.766 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:56.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:56.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:57.076 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:57.077 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:57.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:57.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:57.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:57.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:57.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:57.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:57.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:57.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:57.784 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:57 smithi117 ceph-mon[32835]: pgmap v1023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:57 smithi086 ceph-mon[29229]: pgmap v1023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:57.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:05:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:57 smithi107 ceph-mon[32381]: pgmap v1023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 319 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:58.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:58.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:58.206 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:05:58.206 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:05:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:05:58.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:05:58.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:05:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:58.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:05:58.640 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:05:58.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:05:58.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:58.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:05:59.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:05:59.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:05:59.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:59.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:59.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:05:59.264 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:05:59.264 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:05:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:05:59.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:05:59.458 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:05:59.458 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:05:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:05:59.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:59.574 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:05:59.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:05:59.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:59 smithi086 ceph-mon[29229]: pgmap v1024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:59.767 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:05:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:59.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:05:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:59 smithi117 ceph-mon[32835]: pgmap v1024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:05:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:05:59.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:05:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:59 smithi107 ceph-mon[32381]: pgmap v1024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:05:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:05:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:00.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:00.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:00.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:00.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:00.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:00.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:00.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:00 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:00.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:00 smithi117 ceph-mon[32835]: osdmap e350: 12 total, 3 up, 11 in 2024-08-06T18:06:00.632 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:00 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:00.632 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:00.632 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:00 smithi086 ceph-mon[29229]: osdmap e350: 12 total, 3 up, 11 in 2024-08-06T18:06:00.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:00.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:00.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:00.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:00.884 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:00.884 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:00.884 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:00.885 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:00.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:00.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:00.885 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:00.903 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:00.903 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:00.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:00.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:00.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:00.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:00.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:00.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:00 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1307311480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:00 smithi107 ceph-mon[32381]: osdmap e350: 12 total, 3 up, 11 in 2024-08-06T18:06:00.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:01.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:01.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:01.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:01.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:01.609 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:01.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:01.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:01 smithi086 ceph-mon[29229]: pgmap v1026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:01.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2997940858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:01.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:01 smithi086 ceph-mon[29229]: osdmap e351: 12 total, 3 up, 11 in 2024-08-06T18:06:01.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:01 smithi117 ceph-mon[32835]: pgmap v1026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2997940858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:01 smithi117 ceph-mon[32835]: osdmap e351: 12 total, 3 up, 11 in 2024-08-06T18:06:01.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:01.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:01 smithi107 ceph-mon[32381]: pgmap v1026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:01.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2997940858' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:01.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:01.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:01 smithi107 ceph-mon[32381]: osdmap e351: 12 total, 3 up, 11 in 2024-08-06T18:06:02.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:02.110 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:02.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:02.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:02.326 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:02.326 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:02.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:06:02.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:02.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:02.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:02.813 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:02.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:02.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:02.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:02.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:02.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:03.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:03.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:03.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:03.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:03.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:03.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:03.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:06:03.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:06:03.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:03.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:03 smithi107 ceph-mon[32381]: pgmap v1028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:03.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:03.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:03.751 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:03.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:03.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:03 smithi086 ceph-mon[29229]: pgmap v1028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:03.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:03 smithi117 ceph-mon[32835]: pgmap v1028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:03.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:03.987 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:04.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:04.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:04.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:04.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:04.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:04.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:04.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:04.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:04.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:04.661 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:04.695 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:04.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:04.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:04.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:05.046 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:05.046 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:05.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:05.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:05.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:05.404 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:05.405 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":351,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:01.395873+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:04:54.269240+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:05.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:05.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:05.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:05 smithi086 ceph-mon[29229]: pgmap v1029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:05.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3908126760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:05.651 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:05.651 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:05.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:05 smithi117 ceph-mon[32835]: pgmap v1029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3908126760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:05.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:05 smithi107 ceph-mon[32381]: pgmap v1029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3908126760' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:06.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:06.090 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:06.091 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:06.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:06.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:06.325 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:06.325 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:06.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:06.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:06.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:06.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:06.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:07.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:07.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:07.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:07.512 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:07.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:07.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:07 smithi086 ceph-mon[29229]: pgmap v1030: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:07.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:07.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:07 smithi117 ceph-mon[32835]: pgmap v1030: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:07.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:07 smithi107 ceph-mon[32381]: pgmap v1030: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:08.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:08.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:08.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:08.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:08.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:08.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:08.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:08.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:09.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:09.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:09.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:09.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:09.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:09.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:09.330 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:09.330 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:09.352 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:09.353 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:09.369 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:09.370 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:09.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:09.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:09.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:09.563 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:09.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:09.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:09.834 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:09 smithi117 ceph-mon[32835]: pgmap v1031: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:09.834 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/290151782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:09.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:09 smithi086 ceph-mon[29229]: pgmap v1031: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:09.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/290151782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:09.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:09 smithi107 ceph-mon[32381]: pgmap v1031: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/290151782' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:09.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:09.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:10.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:10.097 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:10.097 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:10.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:10.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:10.358 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:10.358 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:10.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:10.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:10.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:10.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:10.610 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:10.614 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:10.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:10.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:10.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:10.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:10.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:11.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:11.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:11.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:11.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:11.601 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:06:11.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:11.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:11.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:11.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:11.809 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:11.810 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:11.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:11.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:11.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:12.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:12.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:12.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:12.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:12.446 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:12.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:06:12.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:12.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:12.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:12.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:12.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:12.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:12.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:12.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:12 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:12.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:12.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:12 smithi086 ceph-mon[29229]: osdmap e352: 12 total, 3 up, 11 in 2024-08-06T18:06:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:12 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:12 smithi117 ceph-mon[32835]: osdmap e352: 12 total, 3 up, 11 in 2024-08-06T18:06:12.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:12 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2613021439' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:12 smithi107 ceph-mon[32381]: osdmap e352: 12 total, 3 up, 11 in 2024-08-06T18:06:13.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:13.114 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:13.114 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:13.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:13.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:13.389 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:13.389 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:13.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:13.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:13 smithi107 ceph-mon[32381]: pgmap v1034: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:13 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:06:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:13 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:06:13.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:13.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:13 smithi086 ceph-mon[29229]: pgmap v1034: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:13.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:13 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:06:13.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:13 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:06:13.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:06:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:13 smithi117 ceph-mon[32835]: pgmap v1034: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:13 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:06:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:13 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:06:13.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:14.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:14.159 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:14.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:14.169 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:14.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:14.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:14.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:14.618 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:14.618 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:14.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:14.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:14 smithi086 ceph-mon[29229]: osdmap e353: 12 total, 3 up, 11 in 2024-08-06T18:06:14.733 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:14.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:14.789 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:14.789 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":353,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:13.605897+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:14.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:14 smithi117 ceph-mon[32835]: osdmap e353: 12 total, 3 up, 11 in 2024-08-06T18:06:14.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:06:14.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:14 smithi107 ceph-mon[32381]: osdmap e353: 12 total, 3 up, 11 in 2024-08-06T18:06:15.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:15.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:15.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:15.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:15.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:15.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:15 smithi086 ceph-mon[29229]: pgmap v1036: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:15.654 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1354485460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:15.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:15.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:15.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:15 smithi117 ceph-mon[32835]: pgmap v1036: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1354485460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:15.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:15.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:15 smithi107 ceph-mon[32381]: pgmap v1036: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:15.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1354485460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:15.991 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:15.991 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:15.991 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:15.991 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:16.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:16.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:16.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:16.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:16.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:16.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:16.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:16.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:16.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:16.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:16.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:16.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:17.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:17.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:17.346 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:06:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:06:17.346 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:06:17.346 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:06:17.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:17.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:17.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:17.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:17.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:17 smithi086 ceph-mon[29229]: pgmap v1037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:17.800 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:17.800 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:17.800 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:17.800 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:17 smithi117 ceph-mon[32835]: pgmap v1037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:17.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:17.934 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:17 smithi107 ceph-mon[32381]: pgmap v1037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:18.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:18.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:18.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:18.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:18.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:18.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:18.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:18.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:18.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:18.699 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:18.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:18.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:18.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:19.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:19.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:06:19.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:19.374 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:19.374 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:19.375 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:19.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:19.386 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:19.386 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:19.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:19.388 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:19.418 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:19.418 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:19.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:19.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:19.419 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:19.419 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:19.419 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:19.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:19.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:19.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:19.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:19.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:19 smithi086 ceph-mon[29229]: pgmap v1038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2544195293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:19 smithi117 ceph-mon[32835]: pgmap v1038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2544195293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:19.914 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:19.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:19 smithi107 ceph-mon[32381]: pgmap v1038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:19.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2544195293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:20.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:20.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:20.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:20.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:20.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:20.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:20.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:20.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:20.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:20 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:20.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:20.872 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:20.872 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:20 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:20.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:21.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:06:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:06:21.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:06:21.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:06:21.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:06:21.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:20 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:06:21.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:21.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:21.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:21.297 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:21.313 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:21.314 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:21.336 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:21.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:21.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:21.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:21.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:21.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:21.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:21.801 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:21.801 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:21.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:22.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:22.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:22.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:22.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:22.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:22.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:22.807 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:22.807 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:22.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:22.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:23.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:23.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:23.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:06:23.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:23.407 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:23.407 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:23.413 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:06:23.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:23.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:23.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:23 smithi086 ceph-mon[29229]: pgmap v1040: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:23.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:23.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:23 smithi107 ceph-mon[32381]: pgmap v1040: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:23.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:23.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:23.802 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:23 smithi117 ceph-mon[32835]: pgmap v1040: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:23.802 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:23.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:23.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:23.956 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:24.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:24.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:24.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:24.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:24.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:24.475 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:06:24.476 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:06:24.483 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:24.484 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:24.484 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:24 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:24.484 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:24.484 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:24 smithi086 ceph-mon[29229]: osdmap e354: 12 total, 3 up, 11 in 2024-08-06T18:06:24.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:24 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:24 smithi107 ceph-mon[32381]: osdmap e354: 12 total, 3 up, 11 in 2024-08-06T18:06:24.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:24.742 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:24 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:24.743 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/762997281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:24.743 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:24 smithi117 ceph-mon[32835]: osdmap e354: 12 total, 3 up, 11 in 2024-08-06T18:06:24.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:24.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:25.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:25.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:25.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:25.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:25 smithi117 ceph-mon[32835]: pgmap v1042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:25.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:25.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:25 smithi086 ceph-mon[29229]: pgmap v1042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:25.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:25 smithi107 ceph-mon[32381]: pgmap v1042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:25.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:25.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:26.014 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:26.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:26.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:26.084 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:26.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:26.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:26.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:26.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:06:26.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:26.621 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:26.621 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:26.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:26.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:26.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:26.796 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:26.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:26.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:26.870 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":355,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:26.402916+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:26.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:06:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:26.969 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:26.970 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:26.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:27.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:27.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:27.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:27.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:27.551 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:27 smithi107 ceph-mon[32381]: pgmap v1043: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:27.551 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:27 smithi107 ceph-mon[32381]: osdmap e355: 12 total, 3 up, 11 in 2024-08-06T18:06:27.551 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2753209052' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:27.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:27 smithi086 ceph-mon[29229]: pgmap v1043: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:27.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:27 smithi086 ceph-mon[29229]: osdmap e355: 12 total, 3 up, 11 in 2024-08-06T18:06:27.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2753209052' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:27.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:27.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:27.815 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:27.815 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:27 smithi117 ceph-mon[32835]: pgmap v1043: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:27 smithi117 ceph-mon[32835]: osdmap e355: 12 total, 3 up, 11 in 2024-08-06T18:06:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2753209052' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:27.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:28.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:28.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:28.208 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:28.208 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:28.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:28.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:28.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:28.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:28.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:28.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:28.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:28.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:28.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:28.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:28.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:28.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:29.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:29.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:29.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:29.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:29.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:29.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:29.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:29.665 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:29 smithi086 ceph-mon[29229]: pgmap v1045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:29.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:29 smithi107 ceph-mon[32381]: pgmap v1045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:29.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:29.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:29 smithi117 ceph-mon[32835]: pgmap v1045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:29.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:29.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:29.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:30.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:30.195 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:30.196 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:30.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:30.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:30.420 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:06:30.420 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:06:30.427 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:30.451 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:30.451 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:30.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:30.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:30.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:30.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:31.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:31.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:31.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:31.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:06:31.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:31.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:31.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:31 smithi107 ceph-mon[32381]: pgmap v1046: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:31.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:31.562 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:31 smithi086 ceph-mon[29229]: pgmap v1046: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:31.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:31.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:31.641 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:31.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:31.702 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:31 smithi117 ceph-mon[32835]: pgmap v1046: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:31.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:31.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:32.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:32.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:32.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:32.362 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:32.362 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:32.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:32.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:32.650 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:32.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:32.667 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:32.668 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:32.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:32.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:32.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:32.687 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:32.688 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:32.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:32.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:33.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:33.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:33.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:33.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:33.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:33.277 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:33.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:33.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:33.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:33.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:33.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:33.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:33.555 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:33 smithi086 ceph-mon[29229]: pgmap v1047: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:33.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/569547174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:33.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.556 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:33.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:33 smithi107 ceph-mon[32381]: pgmap v1047: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/569547174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:33.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:33.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:33.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:33.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:33 smithi117 ceph-mon[32835]: pgmap v1047: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/569547174' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:33.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:33.990 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:33.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:06:34.014 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:34.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:34.168 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:34.169 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:34.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:34.238 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:34.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:34.423 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:06:34.423 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:34.423 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:34.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:34.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:34.607 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:34.607 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:06:34.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:34.742 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:34.742 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:06:34.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:34.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:34.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:06:34.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:35.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:35.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:35.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:35.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:35.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:35.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:35.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:35.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:35 smithi086 ceph-mon[29229]: pgmap v1048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:35.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:35 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:35.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:06:35.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:35 smithi086 ceph-mon[29229]: osdmap e356: 12 total, 3 up, 11 in 2024-08-06T18:06:35.582 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:35 smithi107 ceph-mon[32381]: pgmap v1048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:35.582 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:35 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:35.582 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:06:35.582 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:35 smithi107 ceph-mon[32381]: osdmap e356: 12 total, 3 up, 11 in 2024-08-06T18:06:35.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:35.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:06:35.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:35 smithi117 ceph-mon[32835]: pgmap v1048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:35 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/216987550' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:06:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:35 smithi117 ceph-mon[32835]: osdmap e356: 12 total, 3 up, 11 in 2024-08-06T18:06:35.930 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:35.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:36.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:36.101 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:36.101 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:36.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:36.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:36.433 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:36.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:36.563 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:36.563 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:36.587 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:36.587 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":357,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:36.405018+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:36.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:36.628 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:36.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:06:36.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:36.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:36.927 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:37.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:37.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:37.158 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:37.158 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:37.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:37.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:37.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:06:37.439 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:37.439 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:37.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:37 smithi086 ceph-mon[29229]: pgmap v1050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:37.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:37 smithi086 ceph-mon[29229]: osdmap e357: 12 total, 3 up, 11 in 2024-08-06T18:06:37.440 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3202927021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:37.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:37 smithi107 ceph-mon[32381]: pgmap v1050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:37 smithi107 ceph-mon[32381]: osdmap e357: 12 total, 3 up, 11 in 2024-08-06T18:06:37.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3202927021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:37.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:37.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:37.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:37 smithi117 ceph-mon[32835]: pgmap v1050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:37 smithi117 ceph-mon[32835]: osdmap e357: 12 total, 3 up, 11 in 2024-08-06T18:06:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3202927021' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:37.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:37.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:37.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:38.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:38.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:38.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:38.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:38.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:38.460 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:38.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:38.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:38.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:38.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:38.940 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:38.941 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:38.941 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:38.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:38.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:38.964 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:38.981 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:38.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:38.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:39.042 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:39.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:06:39.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:39.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:39.290 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:39.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:39.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:39.526 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:39 smithi086 ceph-mon[29229]: pgmap v1052: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:39.526 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3227663403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:39.526 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.526 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:39 smithi117 ceph-mon[32835]: pgmap v1052: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3227663403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:39.675 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:39 smithi107 ceph-mon[32381]: pgmap v1052: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:39.675 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3227663403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:39.675 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.675 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:39.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:39.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:39.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:39.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:40.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:40.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:40.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:40.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:40.404 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:40.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:40.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:40.472 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:40.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:40.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:40.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:40.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:40.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:40.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:40.975 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:41.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:41.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:41.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:41.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:41.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:41.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:41 smithi086 ceph-mon[29229]: pgmap v1053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:41.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:41.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:41 smithi117 ceph-mon[32835]: pgmap v1053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:41.685 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:41 smithi107 ceph-mon[32381]: pgmap v1053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:41.695 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:41.713 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:41.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:06:41.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:41.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:41.981 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:42.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:42.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:42.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:42.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:42.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:06:42.424 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:06:42.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:42.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:42.657 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:42 smithi086 ceph-mon[29229]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:42.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:42 smithi107 ceph-mon[32381]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:42.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:42 smithi117 ceph-mon[32835]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:42.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:43.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:43.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:43.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:43.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:43.474 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:43.474 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:43.474 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:43 smithi086 ceph-mon[29229]: pgmap v1054: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:43.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:43 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:43.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:43 smithi086 ceph-mon[29229]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:43.475 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:43 smithi086 ceph-mon[29229]: osdmap e358: 12 total, 3 up, 11 in 2024-08-06T18:06:43.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:43.597 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:43.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:43 smithi107 ceph-mon[32381]: pgmap v1054: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:43 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:43.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:43 smithi107 ceph-mon[32381]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:43.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:43 smithi107 ceph-mon[32381]: osdmap e358: 12 total, 3 up, 11 in 2024-08-06T18:06:43.748 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:43.748 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":359,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:43.426163+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:43.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:43 smithi117 ceph-mon[32835]: pgmap v1054: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:43 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:43 smithi117 ceph-mon[32835]: from='client.16608 172.21.15.86:0/3534991440' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:43 smithi117 ceph-mon[32835]: osdmap e358: 12 total, 3 up, 11 in 2024-08-06T18:06:43.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:44.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:44.086 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:44.086 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:44.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:44.257 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:44.299 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:44.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:06:44.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:44.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:44.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:44.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:44.547 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:44.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:44.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:44 smithi117 ceph-mon[32835]: osdmap e359: 12 total, 3 up, 11 in 2024-08-06T18:06:44.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3578172564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:44.673 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:44 smithi086 ceph-mon[29229]: osdmap e359: 12 total, 3 up, 11 in 2024-08-06T18:06:44.673 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3578172564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:44.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:44 smithi107 ceph-mon[32381]: osdmap e359: 12 total, 3 up, 11 in 2024-08-06T18:06:44.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3578172564' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:44.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:44.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:45.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:45.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:45.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:45.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:45.436 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:45.436 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:45.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:45.680 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:45 smithi107 ceph-mon[32381]: pgmap v1057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:45.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:45.715 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:45 smithi117 ceph-mon[32835]: pgmap v1057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:45.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:45.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:45 smithi086 ceph-mon[29229]: pgmap v1057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:45.954 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:45.955 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:45.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:46.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:46.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:06:46.199 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:46.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:46.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:46.374 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:46.374 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:46.375 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:46.376 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:46.376 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:46.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:46.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:46.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:46.394 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:46.394 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:46.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:46.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:46.395 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:46.395 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:46.395 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:46.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:46.516 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1438820149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:46.527 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1438820149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:46.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:46.753 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1438820149' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:46.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:46.889 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:46.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:06:46.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:47.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:47.062 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:47.062 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:47.090 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:47.091 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:47.139 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:47.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:47.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:47.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:47.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:47.574 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:47 smithi107 ceph-mon[32381]: pgmap v1058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:47.575 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.575 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.575 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:47 smithi086 ceph-mon[29229]: pgmap v1058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:47.708 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:47 smithi117 ceph-mon[32835]: pgmap v1058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:47.708 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:47.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:47.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:48.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:48.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:48.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:48.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:48.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:48.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:48.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:48.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:48.764 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:48.764 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:48.782 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:06:48.782 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:06:48.782 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:06:48.782 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:06:48.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:48.911 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:49.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:49.033 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:49.033 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:49.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:06:49.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:49.273 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:49.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:49.445 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:06:49.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:49 smithi107 ceph-mon[32381]: pgmap v1059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:49.536 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:49.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:49.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:06:49.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:49 smithi086 ceph-mon[29229]: pgmap v1059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:49.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:49.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:49.700 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:49 smithi117 ceph-mon[32835]: pgmap v1059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:49.700 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:06:49.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:49.844 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:49.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:50.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:50.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:50.269 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:50.269 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:50.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:50.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:50.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:50.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:50.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:50 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:50.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:50.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:50 smithi086 ceph-mon[29229]: osdmap e360: 12 total, 3 up, 11 in 2024-08-06T18:06:50.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:50.630 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:50 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:50.630 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:50.630 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:50 smithi107 ceph-mon[32381]: osdmap e360: 12 total, 3 up, 11 in 2024-08-06T18:06:50.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:50.840 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:50 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:06:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/292259981' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:06:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:50 smithi117 ceph-mon[32835]: osdmap e360: 12 total, 3 up, 11 in 2024-08-06T18:06:50.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:50.952 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:50.952 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:50.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:50.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:51.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:51.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:51.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:51.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:51.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:51.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:51.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:51.574 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:06:51.574 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:06:51.574 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:06:51.574 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:06:51.574 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:06:51.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:51 smithi107 ceph-mon[32381]: pgmap v1061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:51 smithi107 ceph-mon[32381]: osdmap e361: 12 total, 3 up, 11 in 2024-08-06T18:06:51.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:51 smithi086 ceph-mon[29229]: pgmap v1061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:51.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:51 smithi086 ceph-mon[29229]: osdmap e361: 12 total, 3 up, 11 in 2024-08-06T18:06:51.741 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:51.741 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:51.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:51 smithi117 ceph-mon[32835]: pgmap v1061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:51 smithi117 ceph-mon[32835]: osdmap e361: 12 total, 3 up, 11 in 2024-08-06T18:06:51.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:52.006 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:06:52.007 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":361,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:06:51.409452+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:06:52.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:52.163 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:52.163 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:52.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:06:52.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:52.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:52.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:52.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:52.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:06:52.434 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:52.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:52.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:52.629 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/585215337' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:52.630 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:52.630 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:52.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:52.760 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/585215337' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:52.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/585215337' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:06:53.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:53.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:53.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:53.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:53.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:53.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:53.409 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:53.409 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:53.410 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:53.426 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:53.427 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:53.444 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:53.444 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:53.445 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:53.445 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:53.445 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:53.445 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:53.445 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:53.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:53.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:53 smithi117 ceph-mon[32835]: pgmap v1063: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:53.559 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2664983320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:53.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:53.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:53 smithi107 ceph-mon[32381]: pgmap v1063: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:53.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2664983320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:53.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:53 smithi086 ceph-mon[29229]: pgmap v1063: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2664983320' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:06:53.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:53.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:53.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:54.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:54.147 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:06:54.147 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:06:54.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:54.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:54.403 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:06:54.403 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:06:54.451 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:54.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:54.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:54.618 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.618 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.618 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:54.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:54.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:54.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:06:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:06:54.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:06:54.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:55.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:55.184 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:55.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:06:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:06:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:06:55.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:55.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:55.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:55.621 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:55 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:06:55.621 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:55 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:06:55.621 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:55 smithi107 ceph-mon[32381]: pgmap v1064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:55.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:55.679 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:55 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:06:55.679 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:55 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:06:55.680 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:55 smithi117 ceph-mon[32835]: pgmap v1064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:55.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:06:55.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:55 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:06:55.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:55 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:06:55.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:55 smithi086 ceph-mon[29229]: pgmap v1064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:55.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:56.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:56.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:56.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:56.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:06:56.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:56.493 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:06:56.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:56.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:06:56.688 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:56.688 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:56.688 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:56.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:06:56.778 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:06:56.844 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:56.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:56.869 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:56.954 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:06:56.955 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:06:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:06:56.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:57.114 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:57.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:57.355 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:57.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:57.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:57.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:57.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:57 smithi086 ceph-mon[29229]: pgmap v1065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:57 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:57.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:57 smithi086 ceph-mon[29229]: osdmap e362: 12 total, 3 up, 11 in 2024-08-06T18:06:57.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:06:57.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:06:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:57 smithi117 ceph-mon[32835]: pgmap v1065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:57 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:57 smithi117 ceph-mon[32835]: osdmap e362: 12 total, 3 up, 11 in 2024-08-06T18:06:57.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:57.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:57.869 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:06:57.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:57 smithi107 ceph-mon[32381]: pgmap v1065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:57 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:06:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2399341199' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:06:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:57 smithi107 ceph-mon[32381]: osdmap e362: 12 total, 3 up, 11 in 2024-08-06T18:06:58.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:58.094 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:06:58.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:06:58.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:06:58.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:06:58.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:58.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:06:58.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:06:58.881 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:58.881 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:58.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:59.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:06:59.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:59.276 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:06:59.277 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:06:59.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:06:59.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:59.530 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:06:59 smithi086 ceph-mon[29229]: pgmap v1067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:59.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:06:59.618 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:06:59.630 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:06:59.630 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:06:59.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:06:59.649 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:06:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:06:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:06:59.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:06:59.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:06:59.667 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:06:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:06:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:06:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:06:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:06:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:06:59.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:06:59.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:06:59.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:06:59 smithi107 ceph-mon[32381]: pgmap v1067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:59.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:06:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:06:59 smithi117 ceph-mon[32835]: pgmap v1067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:06:59.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:00.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:00.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:00.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:06:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:00.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:00.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:00.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:00.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:00.424 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:00.457 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:00.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:07:00.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:00.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2022021983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:00.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.568 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:00.724 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:07:00.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:00.787 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:00.787 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:00.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2022021983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:00.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:00.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2022021983' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:00.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:01.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:01.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:01.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:01.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:01.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:01 smithi086 ceph-mon[29229]: pgmap v1068: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:01 smithi086 ceph-mon[29229]: osdmap e363: 12 total, 3 up, 11 in 2024-08-06T18:07:01.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:01.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:01 smithi107 ceph-mon[32381]: pgmap v1068: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:01.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:01 smithi107 ceph-mon[32381]: osdmap e363: 12 total, 3 up, 11 in 2024-08-06T18:07:01.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:01 smithi117 ceph-mon[32835]: pgmap v1068: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:01 smithi117 ceph-mon[32835]: osdmap e363: 12 total, 3 up, 11 in 2024-08-06T18:07:01.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:02.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:02.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:02.230 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:02.231 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:02.278 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:02.278 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":363,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:01.412071+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:39.922580+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:02.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:02.392 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:02.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:02.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2806387075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:02.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2806387075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:02.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:02.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2806387075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:02.787 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:02.787 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:02.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:07:02.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:03.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:03.120 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:03.120 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:03.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:03.147 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:03.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:07:03.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:03.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:03.467 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:03.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:03.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:03.686 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:03 smithi117 ceph-mon[32835]: pgmap v1070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:03.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:03.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:03 smithi086 ceph-mon[29229]: pgmap v1070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:03.806 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:03.806 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:03.827 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:03 smithi107 ceph-mon[32381]: pgmap v1070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:03.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:04.057 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:04.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:04.091 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:04.092 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:04.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:04.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:04.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:04.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:04.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:04.511 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:07:04.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:04.612 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:04.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:04.759 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:04.818 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:04.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:04.961 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:04.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:05.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:05.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:05.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:05.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:05.597 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: osdmap e364: 12 total, 3 up, 11 in 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:07:05.598 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:05 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:07:05.639 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:05.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:07:05.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:05.776 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:05.776 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:05.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:05.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:05.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: osdmap e364: 12 total, 3 up, 11 in 2024-08-06T18:07:05.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:05.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:07:05.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:05 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:07:05.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:05.888 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:05.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3825640960' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: osdmap e364: 12 total, 3 up, 11 in 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:07:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:05 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:07:06.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:06.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:06.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:06.319 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:06.320 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:06.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:06.337 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:06.337 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:06.337 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:06.338 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:06.338 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:06.356 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:06.356 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:06.357 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:06.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:06.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:06.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:06 smithi086 ceph-mon[29229]: osdmap e365: 12 total, 3 up, 11 in 2024-08-06T18:07:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3590386277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:06.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:06.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:06 smithi117 ceph-mon[32835]: osdmap e365: 12 total, 3 up, 11 in 2024-08-06T18:07:06.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3590386277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:06.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:06.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:06.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:06.866 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:06 smithi107 ceph-mon[32381]: osdmap e365: 12 total, 3 up, 11 in 2024-08-06T18:07:06.866 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3590386277' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:06.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:07.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:07.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:07.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:07:07.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:07.484 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:07.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:07 smithi086 ceph-mon[29229]: pgmap v1074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:07.706 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:07.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:07.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:07.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:07.785 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:07 smithi117 ceph-mon[32835]: pgmap v1074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:07.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:07 smithi107 ceph-mon[32381]: pgmap v1074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:07.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:07.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:08.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:08.130 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:08.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:07:08.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:08.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:08.381 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:08.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:08.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:08.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:08.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:08.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:08.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:08.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:09.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:09.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:09.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:09.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:09.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:09.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:07:09.722 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:09 smithi117 ceph-mon[32835]: pgmap v1075: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:09.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:09 smithi086 ceph-mon[29229]: pgmap v1075: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:09.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:09.800 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:09 smithi107 ceph-mon[32381]: pgmap v1075: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:09.906 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:09.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:10.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:10.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:10.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:10.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:10.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:10.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:10.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:10.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:07:10.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:10.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:10.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:10.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:10.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:10.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:10.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:10.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:10.796 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:10.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:10.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:11.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:11.197 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:11.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:11.364 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:11.364 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":365,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:05.514302+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:05:49.201825+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:11.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:11.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:11.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1867175366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:11.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:11.663 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1867175366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:11.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:11.794 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1867175366' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:11.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:11.928 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:11.928 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:12.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:12.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:12.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:12.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:12.467 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:12.536 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:07:12.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:12.609 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:07:12.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:07:12.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: osdmap e366: 12 total, 3 up, 11 in 2024-08-06T18:07:12.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:12.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:12.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:12.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:12 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:12.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:07:12.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: osdmap e366: 12 total, 3 up, 11 in 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:12.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:12 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: osdmap e366: 12 total, 3 up, 11 in 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:12.751 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:12.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:12 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:12.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:12.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:13.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:13.179 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:13.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:07:13.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:13.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:13.424 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:13.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:13.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:13.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:13.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: pgmap v1078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: osdmap e367: 12 total, 3 up, 11 in 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:13.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:13.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:13.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:13 smithi107 ceph-mon[32381]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:13.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:13.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: pgmap v1078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:13.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:13.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:13.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: osdmap e367: 12 total, 3 up, 11 in 2024-08-06T18:07:13.838 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:13.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:13.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:13.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:13 smithi117 ceph-mon[32835]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: pgmap v1078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1064391453' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: osdmap e367: 12 total, 3 up, 11 in 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:13 smithi086 ceph-mon[29229]: from='mgr.14148 ' entity='mgr.y' 2024-08-06T18:07:13.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:14.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:14.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:14.291 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:14.292 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:14.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:14.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:14.527 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:14.528 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:14.528 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:14.545 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:14.546 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:14.546 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:14.546 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:14.546 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:14.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:14.563 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:14.564 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:14.564 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:14.564 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:14.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:14.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:14.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:14.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:14.800 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1725793002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:14.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:14.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1725793002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1725793002' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:14.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:15.005 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:15.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:15.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:15.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:15.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:15.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:15.376 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:15.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:15.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:15.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:15.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:15 smithi107 ceph-mon[32381]: pgmap v1080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:15.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.741 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:15.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:07:15.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:15.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:15 smithi086 ceph-mon[29229]: pgmap v1080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:15.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.813 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:15 smithi117 ceph-mon[32835]: pgmap v1080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:15.813 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:15.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:15.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:07:15.991 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:16.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:16.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:16.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:16.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:16.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:16.495 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:16.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:16.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:16.819 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:16.820 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:16.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:16.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:17.078 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:17.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:17.212 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:17.213 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":367,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:12.533157+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:06:12.628080+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:17.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:17.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:07:17.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:17.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:17.576 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:17.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:17.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:17.796 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:17 smithi107 ceph-mon[32381]: pgmap v1081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:17.796 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4229877644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:17 smithi086 ceph-mon[29229]: pgmap v1081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4229877644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:17 smithi117 ceph-mon[32835]: pgmap v1081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4229877644' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:17.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:18.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:18.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:18.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:07:18.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:18.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:18.530 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:18.541 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:18.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:18.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:18.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:18.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:19.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:19.249 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:19.249 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:19.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:19.381 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:19.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:19.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:19.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:19.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:19.667 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:19 smithi107 ceph-mon[32381]: pgmap v1082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:19.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:19.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:19 smithi086 ceph-mon[29229]: pgmap v1082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:19.803 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:19 smithi117 ceph-mon[32835]: pgmap v1082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:19.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:20.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:20.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:20.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:20.165 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:20.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:20.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:20.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:20.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:20.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:20.558 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:07:20.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:20.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:20 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:07:20.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:20 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:07:20.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:20 smithi117 ceph-mon[32835]: osdmap e368: 12 total, 3 up, 11 in 2024-08-06T18:07:20.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:20.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:20.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:20 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:07:20.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:20 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:07:20.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:20 smithi086 ceph-mon[29229]: osdmap e368: 12 total, 3 up, 11 in 2024-08-06T18:07:20.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:20.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:20.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:07:20.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:20 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:07:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:20 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:07:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:20 smithi107 ceph-mon[32381]: osdmap e368: 12 total, 3 up, 11 in 2024-08-06T18:07:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:20.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:21.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:21.067 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:21.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:21.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:21.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:21.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:21.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:21.565 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:21.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:21 smithi086 ceph-mon[29229]: pgmap v1084: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:21.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:21 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:21.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:21.565 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:21 smithi086 ceph-mon[29229]: osdmap e369: 12 total, 3 up, 11 in 2024-08-06T18:07:21.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:21.663 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:21.663 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:21.664 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:21.682 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:21.683 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:21.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:21 smithi107 ceph-mon[32381]: pgmap v1084: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:21.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:21 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:21.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:21.684 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:21 smithi107 ceph-mon[32381]: osdmap e369: 12 total, 3 up, 11 in 2024-08-06T18:07:21.702 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:21.702 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:21.703 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:21.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:21.833 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:21 smithi117 ceph-mon[32835]: pgmap v1084: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:21.833 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:21 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:21.833 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/71747978' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:21.833 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:21 smithi117 ceph-mon[32835]: osdmap e369: 12 total, 3 up, 11 in 2024-08-06T18:07:21.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:22.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:22.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:22.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:22.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:22.497 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:22.497 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:22.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:22.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1961246487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:22.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.595 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:22.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1961246487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:22.743 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:22.744 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:22.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1961246487' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:22.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:22.804 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:22.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:22.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:23.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:23.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:07:23.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:23.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:23.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:07:23.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:23.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:23.697 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:23.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:23.810 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:23.811 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:23.811 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:23.811 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:23 smithi107 ceph-mon[32381]: pgmap v1086: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:23.817 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:23.817 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:23 smithi117 ceph-mon[32835]: pgmap v1086: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:23.827 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:23.827 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:23 smithi086 ceph-mon[29229]: pgmap v1086: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:23.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:24.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:24.073 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:24.073 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:24.180 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:24.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:24.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:24.340 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:24.340 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":369,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:20.555873+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:24.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:24.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:07:24.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:24.602 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3750587513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:24.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:24.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3750587513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:24.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3750587513' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:24.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:24.865 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:24.866 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:24.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:25.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:25.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:25.284 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:25.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:25.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:25.574 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:25.574 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:25.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:25.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:25.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:25 smithi086 ceph-mon[29229]: pgmap v1087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:25 smithi117 ceph-mon[32835]: pgmap v1087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:25.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:25.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:25 smithi107 ceph-mon[32381]: pgmap v1087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:25.989 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:25.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:07:26.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:26.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:26.181 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:26.232 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:26.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:26.426 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:26.427 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:26.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:26.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:26 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:07:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:26 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:07:26.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:26.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:26 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:07:26.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:26 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:07:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:26.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:26.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:26.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:26 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:07:26.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:26 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:07:26.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:27.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:27.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:27.352 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:27.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:27.584 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:07:27.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:27.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:27.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:27 smithi107 ceph-mon[32381]: pgmap v1088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:27.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:27.775 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:27.775 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:27.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:27 smithi086 ceph-mon[29229]: pgmap v1088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:27.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:27 smithi117 ceph-mon[32835]: pgmap v1088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:27.881 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:27.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:28.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:28.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:28.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:28.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:28.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:28.436 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:28.436 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:28.528 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:28.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:07:28.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:28.681 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:28.706 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:28 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:07:28.706 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:28.706 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:28 smithi107 ceph-mon[32381]: osdmap e370: 12 total, 3 up, 11 in 2024-08-06T18:07:28.706 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:28.706 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:28.773 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:28.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:28.861 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:28 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:07:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:28 smithi086 ceph-mon[29229]: osdmap e370: 12 total, 3 up, 11 in 2024-08-06T18:07:28.869 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:28.869 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:28.869 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:28 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:07:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2927256892' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:28 smithi117 ceph-mon[32835]: osdmap e370: 12 total, 3 up, 11 in 2024-08-06T18:07:28.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:28.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:28.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:28.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:28.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:28.906 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:28.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:28.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:29.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:29.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:29.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:29.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:29.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:29.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:29.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:29.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:29.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:29.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:29 smithi086 ceph-mon[29229]: pgmap v1090: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:29.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4266743435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:29.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.851 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:29.851 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:29.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:29 smithi117 ceph-mon[32835]: pgmap v1090: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:29.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4266743435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:29.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:29.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:29 smithi107 ceph-mon[32381]: pgmap v1090: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:29.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4266743435' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:29.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:29.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:30.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:30.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:30.362 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:30.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:30.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:30.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:30.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:07:30.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:30.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:30.729 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:30.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:30.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:30.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:30.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:31.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:31.067 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:31.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:07:31.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:31.203 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:31.315 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:07:31.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:31.334 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:31.334 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:31.401 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:31.401 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":370,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:27.581595+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:31.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:31.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:31.594 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:31.594 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:31 smithi117 ceph-mon[32835]: pgmap v1091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2128635093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:31.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:31 smithi117 ceph-mon[32835]: osdmap e371: 12 total, 3 up, 11 in 2024-08-06T18:07:31.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:31.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:31.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:07:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:31 smithi086 ceph-mon[29229]: pgmap v1091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2128635093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:31 smithi086 ceph-mon[29229]: osdmap e371: 12 total, 3 up, 11 in 2024-08-06T18:07:31.873 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:31.873 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:31.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:31 smithi107 ceph-mon[32381]: pgmap v1091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:31.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2128635093' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:31.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:31 smithi107 ceph-mon[32381]: osdmap e371: 12 total, 3 up, 11 in 2024-08-06T18:07:31.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:32.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:32.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:32.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:32.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:32.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:32.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:32.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:32.942 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:33.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:33.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:33.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:33.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:33.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:33.608 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:33.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:07:33.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:33.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:33 smithi086 ceph-mon[29229]: pgmap v1093: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:33.721 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:33.721 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:33.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:33.861 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:33 smithi107 ceph-mon[32381]: pgmap v1093: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:33 smithi117 ceph-mon[32835]: pgmap v1093: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:33.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:34.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:34.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:34.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:34.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:34.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:34.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:34.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:34.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:34.602 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:07:34.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:34.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:34.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:34.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:34.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:34.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:07:35.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:35.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:35.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:35.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:35.499 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:35.501 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:35.515 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:35.516 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:35.517 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:35.534 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:35.534 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:35.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:35.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:35.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:35.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:35.836 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:35 smithi086 ceph-mon[29229]: pgmap v1094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:35.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:35 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:35.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:35.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:35 smithi086 ceph-mon[29229]: osdmap e372: 12 total, 3 up, 11 in 2024-08-06T18:07:35.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2229814544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:35.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:35 smithi117 ceph-mon[32835]: pgmap v1094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:35 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:35 smithi117 ceph-mon[32835]: osdmap e372: 12 total, 3 up, 11 in 2024-08-06T18:07:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2229814544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:35.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:35 smithi107 ceph-mon[32381]: pgmap v1094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:35.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:35 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:35.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/354012347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:07:35.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:35 smithi107 ceph-mon[32381]: osdmap e372: 12 total, 3 up, 11 in 2024-08-06T18:07:35.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2229814544' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:35.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:36.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:36.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:36.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:36.256 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:36.256 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:36.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:36.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:36.435 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:36.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:36.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:36 smithi107 ceph-mon[32381]: osdmap e373: 12 total, 3 up, 11 in 2024-08-06T18:07:36.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:36.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:36 smithi117 ceph-mon[32835]: osdmap e373: 12 total, 3 up, 11 in 2024-08-06T18:07:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:36 smithi086 ceph-mon[29229]: osdmap e373: 12 total, 3 up, 11 in 2024-08-06T18:07:36.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:36.941 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:37.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:37.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:37.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:37.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:37.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:37.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:37.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:37 smithi086 ceph-mon[29229]: pgmap v1096: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:37.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:37 smithi117 ceph-mon[32835]: pgmap v1096: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:37 smithi107 ceph-mon[32381]: pgmap v1096: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:38.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:38.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:38.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:38.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:38.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:07:38.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:38.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:38.433 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:38.433 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:38.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:38.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:38.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:38.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:07:38.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:39.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:39.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:39.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:39.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:39.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:39.475 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:39.475 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":373,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:36.421466+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:39.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:39.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:39.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:39.613 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:39.613 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:39.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:39.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:39.799 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:39.799 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:39.799 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:39 smithi107 ceph-mon[32381]: pgmap v1098: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:39.799 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3929632272' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:39 smithi086 ceph-mon[29229]: pgmap v1098: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3929632272' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:39.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:39.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:39 smithi117 ceph-mon[32835]: pgmap v1098: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3929632272' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:39.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:40.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:40.141 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:40.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:40.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:40.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:40.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:40.709 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:40.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:40.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:40.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:40.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:40.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:07:40.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:40.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:07:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:07:40.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:41.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:41.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:41.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:41.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:41.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:41.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:41.817 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:41.817 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:41.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:42.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:42.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:42.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:42.400 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:42.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:42.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:42.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:42.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:43.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:43.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:43.194 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:43.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:07:43.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:43.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:43.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:43.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:43.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:43.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:43 smithi117 ceph-mon[32835]: pgmap v1100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:43.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:43.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:43.697 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:43 smithi086 ceph-mon[29229]: pgmap v1100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:43.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:43.698 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:43.700 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:43 smithi107 ceph-mon[32381]: pgmap v1100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:43.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:43.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:43.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:43.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:44.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:44.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:44.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:44.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:44.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:44.755 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:44.756 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:44.756 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:44.772 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:44.773 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:44.773 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:44.773 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:44.773 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:44.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:44.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:44.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:44.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:44.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:44.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:44.792 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:44.792 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:44.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:45.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:45.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:45.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:45.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:45.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:45.432 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:07:45.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:45.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:45 smithi086 ceph-mon[29229]: pgmap v1101: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3433393209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:45 smithi117 ceph-mon[32835]: pgmap v1101: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3433393209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:45.655 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:45.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:45.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:45.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:45 smithi107 ceph-mon[32381]: pgmap v1101: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3433393209' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:07:45.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:45.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:45.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:46.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:46.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:46.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:46.396 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:46.418 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:46.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:46.558 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:46.559 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:46 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:46.559 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:46.559 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:46 smithi086 ceph-mon[29229]: osdmap e374: 12 total, 3 up, 11 in 2024-08-06T18:07:46.559 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:46 smithi086 ceph-mon[29229]: osdmap e375: 12 total, 3 up, 11 in 2024-08-06T18:07:46.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:46.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:46 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:46 smithi107 ceph-mon[32381]: osdmap e374: 12 total, 3 up, 11 in 2024-08-06T18:07:46.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:46 smithi107 ceph-mon[32381]: osdmap e375: 12 total, 3 up, 11 in 2024-08-06T18:07:46.797 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:46.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:46.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:46 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:46.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3933407381' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:07:46.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:46 smithi117 ceph-mon[32835]: osdmap e374: 12 total, 3 up, 11 in 2024-08-06T18:07:46.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:46 smithi117 ceph-mon[32835]: osdmap e375: 12 total, 3 up, 11 in 2024-08-06T18:07:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:46.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:47.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:47.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:47.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:47.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:47.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:47.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:47.504 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:47 smithi107 ceph-mon[32381]: pgmap v1103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:47.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:47.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:47 smithi117 ceph-mon[32835]: pgmap v1103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:47.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:07:47.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:47.770 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:47 smithi086 ceph-mon[29229]: pgmap v1103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:47.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:48.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:48.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:48.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:48.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:07:48.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:48.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:48.570 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:07:48.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:48.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:48.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:48.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:48.571 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":375,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:46.424514+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:07:48.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:48.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:48.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:49.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:49.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:49.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:49.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:49.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:49.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:49 smithi086 ceph-mon[29229]: pgmap v1105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:49.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3453822320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:49 smithi117 ceph-mon[32835]: pgmap v1105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3453822320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:49.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:49.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:49 smithi107 ceph-mon[32381]: pgmap v1105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:49.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3453822320' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:07:49.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:49.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:50.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:50.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:50.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:50.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:50.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:50.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:50.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:50.662 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:50.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:50.797 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:50.863 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:50.867 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:50.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:51.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:51.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:51.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:51.341 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:51.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:07:51.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:51.569 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:51 smithi086 ceph-mon[29229]: pgmap v1106: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:51.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:51.611 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:51 smithi117 ceph-mon[32835]: pgmap v1106: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:51 smithi107 ceph-mon[32381]: pgmap v1106: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:51.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:51.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:51.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:51.998 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:52.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:52.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:52.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:52.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:52.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:52.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:07:52.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:52.803 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:52.804 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:52.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:52.958 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:52.959 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:53.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:53.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:53.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:53.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:53.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:53.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:53.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:53.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:53 smithi117 ceph-mon[32835]: pgmap v1107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:53.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:53 smithi107 ceph-mon[32381]: pgmap v1107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:53.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:53.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:53 smithi086 ceph-mon[29229]: pgmap v1107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:53.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:54.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:54.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:54.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:54.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:54.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:54.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:54.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:54.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:54.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:54.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:54.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:54.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:54.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:55.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:55.140 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:07:55.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:55.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:55.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:55.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:55.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:55.497 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:55 smithi086 ceph-mon[29229]: pgmap v1108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:55.497 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:55.497 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:55.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:55.705 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:07:55.706 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:07:55.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:07:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:55 smithi107 ceph-mon[32381]: pgmap v1108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:55.732 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:07:55.733 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:07:55.733 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:07:55.733 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:07:55.733 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:07:55.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:55.750 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:07:55.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:07:55.750 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:55.750 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:55.751 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:07:55.751 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:07:55.751 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:07:55.751 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:07:55.751 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:07:55.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:55 smithi117 ceph-mon[32835]: pgmap v1108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:55.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:56.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:56.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:56.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:56.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:56.463 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:07:56.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:07:56.576 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3553597407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:56.576 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.576 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.576 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:56.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:56.712 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3553597407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:56.712 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.712 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.712 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:56.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:56.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3553597407' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:07:56.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:56.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:07:56.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:07:57.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:57.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:57.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:57.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:57.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:57.464 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:57.465 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:57.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:57.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:57.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:57.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:57 smithi086 ceph-mon[29229]: pgmap v1109: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:57.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:57 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:57.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:57.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:57 smithi086 ceph-mon[29229]: osdmap e376: 12 total, 3 up, 11 in 2024-08-06T18:07:57.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:07:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:57 smithi117 ceph-mon[32835]: pgmap v1109: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:57 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:57 smithi117 ceph-mon[32835]: osdmap e376: 12 total, 3 up, 11 in 2024-08-06T18:07:57.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:57.928 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:57.928 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:57.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:07:57.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:57 smithi107 ceph-mon[32381]: pgmap v1109: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:57.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:57 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:07:57.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2968476047' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:07:57.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:57 smithi107 ceph-mon[32381]: osdmap e376: 12 total, 3 up, 11 in 2024-08-06T18:07:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:07:57.977 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:07:58.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:58.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:07:58.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:58.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:58.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:58.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:58.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:58.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:58.490 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:07:58.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:07:58.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:58.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:58.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:59.001 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:59.001 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:59.001 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:59.001 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:59.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:07:59.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:07:59.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:07:59.265 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:07:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:07:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:07:59.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:07:59.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:07:59.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:07:59.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:07:59 smithi086 ceph-mon[29229]: pgmap v1111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:59.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:07:59 smithi117 ceph-mon[32835]: pgmap v1111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:59.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:07:59.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:07:59.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:07:59.901 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:07:59.901 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:07:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:07:59.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:07:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:07:59 smithi107 ceph-mon[32381]: pgmap v1111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:07:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:07:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:07:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:07:59.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:08:00.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:00.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:00.320 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:00.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:00.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:00.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:00.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:00.482 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:00.482 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":376,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:07:56.460545+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:04.925206+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:00.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:00.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:00.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:01.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:01.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:01.218 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:01.225 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:01.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:01.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:01.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:01.571 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:01 smithi086 ceph-mon[29229]: pgmap v1112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:01.571 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3658343622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:01.571 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:01 smithi086 ceph-mon[29229]: osdmap e377: 12 total, 3 up, 11 in 2024-08-06T18:08:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:01 smithi117 ceph-mon[32835]: pgmap v1112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3658343622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:01 smithi117 ceph-mon[32835]: osdmap e377: 12 total, 3 up, 11 in 2024-08-06T18:08:01.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:01.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:01.836 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:01 smithi107 ceph-mon[32381]: pgmap v1112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:01.837 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3658343622' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:01.837 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:01 smithi107 ceph-mon[32381]: osdmap e377: 12 total, 3 up, 11 in 2024-08-06T18:08:01.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:01.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:01.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:02.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:02.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:02.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:02.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:02.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:02.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:02 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:08:02.515 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:02 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:08:02.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:02.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:02.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:02 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:08:02.777 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:02 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:08:02.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:02 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:08:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:02 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:08:03.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:03.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:03.160 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:03.160 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:03.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:03.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:03.477 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:03.477 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:03.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:03.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:03.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:03 smithi107 ceph-mon[32381]: pgmap v1114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:03.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:03.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:03 smithi086 ceph-mon[29229]: pgmap v1114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:03 smithi117 ceph-mon[32835]: pgmap v1114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:03.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:04.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:04.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:04.256 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:04.256 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:04.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:04.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:04.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:04.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:04.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:04.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:04.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:04.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:04.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:05.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:05.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:05.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:05.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:05.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:05.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:05.443 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:05.443 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:05.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:05.484 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:08:05.488 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:05.488 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:05.489 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:05.511 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:05.512 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:05.530 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:05.531 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:05.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:05.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:05.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:05.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:05 smithi086 ceph-mon[29229]: pgmap v1115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:05.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:05.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/933408420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:05.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:05 smithi117 ceph-mon[32835]: pgmap v1115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/933408420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:05.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:05 smithi107 ceph-mon[32381]: pgmap v1115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:05.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:05.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/933408420' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:05.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:06.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:06.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:06.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:06.182 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:08:06.182 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:08:06.182 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:08:06.182 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:08:06.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:06.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:06.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: osdmap e378: 12 total, 3 up, 11 in 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.548 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:06 smithi107 ceph-mon[32381]: osdmap e379: 12 total, 3 up, 11 in 2024-08-06T18:08:06.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:06.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: osdmap e378: 12 total, 3 up, 11 in 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.813 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:06 smithi086 ceph-mon[29229]: osdmap e379: 12 total, 3 up, 11 in 2024-08-06T18:08:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2196025499' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: osdmap e378: 12 total, 3 up, 11 in 2024-08-06T18:08:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:06.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:06 smithi117 ceph-mon[32835]: osdmap e379: 12 total, 3 up, 11 in 2024-08-06T18:08:06.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:08:06.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:07.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:07.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:07.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:07.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:07.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:07.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:07.545 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:07.546 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:07.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:07.597 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:07.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:07.752 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:07 smithi107 ceph-mon[32381]: pgmap v1117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:07.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:07.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:07 smithi086 ceph-mon[29229]: pgmap v1117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:07.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:07.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:07 smithi117 ceph-mon[32835]: pgmap v1117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 320 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:07.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:08.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:08.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:08.273 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:08.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:08.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:08.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:08.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:08.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:08.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:09.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:09.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:09.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:09.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:09.214 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:09.214 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:09.258 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:09.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:09.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:09.493 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:09.493 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:09.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:08:09.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:09.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:09.743 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:09 smithi086 ceph-mon[29229]: pgmap v1119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:09.743 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:09 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:08:09.743 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:09 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:08:09.743 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:09 smithi086 ceph-mon[29229]: osdmap e380: 12 total, 3 up, 11 in 2024-08-06T18:08:09.746 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:09 smithi117 ceph-mon[32835]: pgmap v1119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:09.747 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:09 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:08:09.747 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:09 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:08:09.747 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:09 smithi117 ceph-mon[32835]: osdmap e380: 12 total, 3 up, 11 in 2024-08-06T18:08:09.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:09 smithi107 ceph-mon[32381]: pgmap v1119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:09 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:08:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:09 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:08:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:09 smithi107 ceph-mon[32381]: osdmap e380: 12 total, 3 up, 11 in 2024-08-06T18:08:09.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:10.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:08:10.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:10.241 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:10.241 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":380,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:08:08.489391+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:19.245002+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:10.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:10.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:10.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:10.558 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1732907466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:10.558 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:10.558 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:10.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:10.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1732907466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:10.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:10.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1732907466' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:10.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:11.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:11.162 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:11.162 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:11.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:11.310 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:11.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:11.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:11.584 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:08:11.584 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:08:11.585 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:08:11.585 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:08:11.585 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:08:11.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:11.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:11.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:11 smithi107 ceph-mon[32381]: pgmap v1121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:11.763 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:11 smithi107 ceph-mon[32381]: osdmap e381: 12 total, 3 up, 11 in 2024-08-06T18:08:11.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:11 smithi086 ceph-mon[29229]: pgmap v1121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:11.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:11 smithi086 ceph-mon[29229]: osdmap e381: 12 total, 3 up, 11 in 2024-08-06T18:08:11.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:11.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:11.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:11 smithi117 ceph-mon[32835]: pgmap v1121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:11 smithi117 ceph-mon[32835]: osdmap e381: 12 total, 3 up, 11 in 2024-08-06T18:08:12.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:12.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:12.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:12.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:12.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:12.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:12.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:12.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:12.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:12.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:13.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:13.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:13.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:13.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:13.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:08:13.551 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:13.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:13.638 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:13 smithi086 ceph-mon[29229]: pgmap v1123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:13.638 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:13.638 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:13.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:13.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:13 smithi107 ceph-mon[32381]: pgmap v1123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:13.772 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:13.772 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:13 smithi117 ceph-mon[32835]: pgmap v1123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:13.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:14.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:14.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:14.289 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:14.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:14.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:14.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:14.552 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:14.552 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:14.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:14.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:14.569 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:14.570 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:14.570 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:14.570 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:14.570 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:14.588 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:14.588 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:14.588 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:14.589 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:14.589 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:14.589 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:14.590 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:14.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:14.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:14.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:15.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:15.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:15.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:15.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:15.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:15.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:15.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:15.421 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:15.421 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:15.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:15.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:15.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:15 smithi086 ceph-mon[29229]: pgmap v1124: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:15.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1947032911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:15.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:15.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:15 smithi107 ceph-mon[32381]: pgmap v1124: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1947032911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:15.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:15 smithi117 ceph-mon[32835]: pgmap v1124: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:15.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1947032911' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:15.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:15.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:15.944 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:15.945 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:15.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:16.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:16.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:16.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:16.429 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:16.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:16.537 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:16.537 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:16.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:16.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:16.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:16.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:16.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:16.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:16.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:16.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:17.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:17.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:17.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:17.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:08:17.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:17.511 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:08:17.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:17.667 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:17 smithi107 ceph-mon[32381]: pgmap v1125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:17.667 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:08:17.702 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:17 smithi086 ceph-mon[29229]: pgmap v1125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:17.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:08:17.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:17 smithi117 ceph-mon[32835]: pgmap v1125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:08:17.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:17.942 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:17.954 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:18.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:18.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:18.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:18.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:18.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:18.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:18.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:18.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:18.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:18 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:08:18.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:18 smithi086 ceph-mon[29229]: osdmap e382: 12 total, 3 up, 11 in 2024-08-06T18:08:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:18 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:08:18.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:18 smithi117 ceph-mon[32835]: osdmap e382: 12 total, 3 up, 11 in 2024-08-06T18:08:18.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:18.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:18.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:18 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3357453782' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:08:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:18 smithi107 ceph-mon[32381]: osdmap e382: 12 total, 3 up, 11 in 2024-08-06T18:08:18.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:18.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:18.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:18.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:18.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:19.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:19.205 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:19.205 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:19.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:19.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:19.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:19.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:19.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:19 smithi107 ceph-mon[32381]: pgmap v1127: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:19.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:19.813 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:19 smithi117 ceph-mon[32835]: pgmap v1127: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:19.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:19 smithi086 ceph-mon[29229]: pgmap v1127: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:19.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:20.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:20.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:20.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:20.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:20.209 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:08:20.210 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:08:20.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:20.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:20.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:20.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:20.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:20.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:20.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:20.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:08:20.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:08:21.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:21.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:21.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:21.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:21.445 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:21.446 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:21.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:21.581 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:21 smithi117 ceph-mon[32835]: pgmap v1128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:21.581 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:08:21.581 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:08:21.581 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:21 smithi117 ceph-mon[32835]: osdmap e383: 12 total, 3 up, 11 in 2024-08-06T18:08:21.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:21 smithi086 ceph-mon[29229]: pgmap v1128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:08:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:08:21.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:21 smithi086 ceph-mon[29229]: osdmap e383: 12 total, 3 up, 11 in 2024-08-06T18:08:21.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:21 smithi107 ceph-mon[32381]: pgmap v1128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:08:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:08:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:21 smithi107 ceph-mon[32381]: osdmap e383: 12 total, 3 up, 11 in 2024-08-06T18:08:21.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:21.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:21.959 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:21.959 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:22.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:22.184 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:22.192 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:22.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:22.211 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:22.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:22.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:22.229 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:22.230 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:22.230 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:22.230 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:22.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:22.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:22.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:22.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:22.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:22.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:22.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:22.785 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3791298293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:22.786 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:22.786 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:22.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3791298293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3791298293' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:22.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:23.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:23.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:23.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:23.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:23.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:23.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:23.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:23.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:23.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:23 smithi117 ceph-mon[32835]: pgmap v1130: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:23.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:23 smithi107 ceph-mon[32381]: pgmap v1130: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:23.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.715 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.724 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:23 smithi086 ceph-mon[29229]: pgmap v1130: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:23.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:23.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:23.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:24.023 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:24.023 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:24.072 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:24.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:24.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:24.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:24.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:24.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:24.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:24.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:24.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:24.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:24.915 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:25.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:25.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:25.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:25.338 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:25.338 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:25.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:25.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:25.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:25 smithi086 ceph-mon[29229]: pgmap v1131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:25.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:25 smithi117 ceph-mon[32835]: pgmap v1131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:25.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:25.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:25 smithi107 ceph-mon[32381]: pgmap v1131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:25.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:26.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:26.212 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:26.212 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":383,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:08:20.519894+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:07:11.110714+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:26.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:26.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:26.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:26.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:26.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:26.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1583786477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1583786477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:26.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:26.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:08:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1583786477' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:27.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:27.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:27.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:27.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:27.470 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:27.470 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:27.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:27.627 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:27 smithi117 ceph-mon[32835]: pgmap v1132: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:27.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:27.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:27 smithi086 ceph-mon[29229]: pgmap v1132: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:27.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:27 smithi107 ceph-mon[32381]: pgmap v1132: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:27.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:27.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:27.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:28.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:28.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:28.381 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:28.393 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:08:28.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:28.546 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:08:28.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:28.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:28.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:28.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:28.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:28.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:28.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:29.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:29.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:29.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:29.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:29.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:29.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:29.466 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:29.466 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:29.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:29.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:29 smithi107 ceph-mon[32381]: pgmap v1133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:29 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:29 smithi107 ceph-mon[32381]: osdmap e384: 12 total, 3 up, 11 in 2024-08-06T18:08:29.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:29.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:29 smithi086 ceph-mon[29229]: pgmap v1133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:29.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:29 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:29.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:29.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:29 smithi086 ceph-mon[29229]: osdmap e384: 12 total, 3 up, 11 in 2024-08-06T18:08:29.794 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:08:29.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:08:29.840 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:29.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:29 smithi117 ceph-mon[32835]: pgmap v1133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:29 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3789181102' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:29 smithi117 ceph-mon[32835]: osdmap e384: 12 total, 3 up, 11 in 2024-08-06T18:08:29.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:30.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:30.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:30.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:30.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:30.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:30.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:30.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:30.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:30.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:30.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:30 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:08:30.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:30 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:08:30.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:30 smithi107 ceph-mon[32381]: osdmap e385: 12 total, 3 up, 11 in 2024-08-06T18:08:30.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:30 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:08:30.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:30 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:08:30.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:30 smithi086 ceph-mon[29229]: osdmap e385: 12 total, 3 up, 11 in 2024-08-06T18:08:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:30 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:08:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:30 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:08:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:30 smithi117 ceph-mon[32835]: osdmap e385: 12 total, 3 up, 11 in 2024-08-06T18:08:30.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:30.985 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:31.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:31.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:31.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:31.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:08:31.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:31.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:31.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:31.763 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:31.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:31 smithi086 ceph-mon[29229]: pgmap v1136: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:31 smithi117 ceph-mon[32835]: pgmap v1136: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:31.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:31 smithi107 ceph-mon[32381]: pgmap v1136: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:31.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:31.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:32.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:32.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:32.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:32.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:32.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:32.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:32.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:32.833 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:32 smithi107 ceph-mon[32381]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.833 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:32 smithi107 ceph-mon[32381]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:32 smithi086 ceph-mon[29229]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:32 smithi086 ceph-mon[29229]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:32 smithi117 ceph-mon[32835]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:32 smithi117 ceph-mon[32835]: from='client.16836 172.21.15.86:0/180194936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:32.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:33.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:33.098 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:33.098 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:33.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:33.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:33.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:33.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:33 smithi117 ceph-mon[32835]: pgmap v1137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:33.760 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:33 smithi086 ceph-mon[29229]: pgmap v1137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:33.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:33.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:33.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:33.838 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:33.838 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:33.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:33.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:33 smithi107 ceph-mon[32381]: pgmap v1137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:33.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:33.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:34.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:34.025 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:34.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:34.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:34.406 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:34.406 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:34.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:34.438 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:34.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:34.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:34.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:34.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:34.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:34.845 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:08:34.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:35.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:35.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:35.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:35.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:35.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:35.639 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:35 smithi086 ceph-mon[29229]: pgmap v1138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:35.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:35.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:35.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:35 smithi117 ceph-mon[32835]: pgmap v1138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:35.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:35.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:35 smithi107 ceph-mon[32381]: pgmap v1138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:36.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:36.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:36.205 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:36.205 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:36.206 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:08:36.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:08:36.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:36.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:36.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:08:36.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:36.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:36.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:36.897 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:36.898 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":385,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:08:29.548247+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:36.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:37.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:37.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:37.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:37.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:37.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:37.716 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:37.717 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:37.717 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:37 smithi086 ceph-mon[29229]: pgmap v1139: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:37.717 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1459923534' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:37.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:37 smithi117 ceph-mon[32835]: pgmap v1139: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1459923534' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:37.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:37 smithi107 ceph-mon[32381]: pgmap v1139: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1459923534' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:37.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:37.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:38.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:38.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:38.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:38.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:38.312 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:38.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:38.350 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:38.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:38.575 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:08:38.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:38.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:38.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:38.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:08:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:08:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:08:38.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:39.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:39.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:39.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:39.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:39.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:39.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:39.663 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:39.733 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:39.733 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:39.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:39 smithi086 ceph-mon[29229]: pgmap v1140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:39.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:39 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:39.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:08:39.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:39 smithi086 ceph-mon[29229]: osdmap e386: 12 total, 3 up, 11 in 2024-08-06T18:08:39.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:39 smithi117 ceph-mon[32835]: pgmap v1140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:39 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:08:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:39 smithi117 ceph-mon[32835]: osdmap e386: 12 total, 3 up, 11 in 2024-08-06T18:08:39.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:39.955 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:39 smithi107 ceph-mon[32381]: pgmap v1140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:39.955 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:39 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:08:39.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3880461901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:08:39.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:39 smithi107 ceph-mon[32381]: osdmap e386: 12 total, 3 up, 11 in 2024-08-06T18:08:40.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:40.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:40.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:40.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:40.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:40.613 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:40 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:08:40.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:40.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:40 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:08:40.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:40.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:40.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:40 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:08:40.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:41.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:41.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:41.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:41.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:41.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:41.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:41.661 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:41 smithi086 ceph-mon[29229]: pgmap v1142: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:41.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:41 smithi086 ceph-mon[29229]: osdmap e387: 12 total, 3 up, 11 in 2024-08-06T18:08:41.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.662 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:41 smithi107 ceph-mon[32381]: pgmap v1142: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:41 smithi107 ceph-mon[32381]: osdmap e387: 12 total, 3 up, 11 in 2024-08-06T18:08:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:41 smithi117 ceph-mon[32835]: pgmap v1142: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:41 smithi117 ceph-mon[32835]: osdmap e387: 12 total, 3 up, 11 in 2024-08-06T18:08:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:41.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:41.959 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:08:42.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:42.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:42.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:42.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:42.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:42.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:08:42.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:42.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:42.626 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:08:42.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:08:42.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:42.811 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:42.812 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:42.812 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:42.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:42.831 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:42.832 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:42.833 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:42.833 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:42.833 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:42.833 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:42.851 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:42.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:42.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:42.857 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:42.857 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:42.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:42.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:42.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:42.859 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:42.859 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:42.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:43.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:43.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:43.254 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:43.254 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:43.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:43.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:43.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:43.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:43.582 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:43.582 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:43.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:43.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:43.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:43 smithi086 ceph-mon[29229]: pgmap v1144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:43.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/261607035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:43.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:43.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:43 smithi117 ceph-mon[32835]: pgmap v1144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:43.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/261607035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:43.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:43.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:43.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:43 smithi107 ceph-mon[32381]: pgmap v1144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/261607035' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:44.028 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:44.028 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:44.029 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:44.029 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:44.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:44.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:44.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:44.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:44.570 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:44.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:44.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:44.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:44.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:44.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:45.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:45.136 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:45.137 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:45.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:45.268 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:45.269 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:45.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:45.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:45.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:08:45.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:45.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:45 smithi086 ceph-mon[29229]: pgmap v1145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:45.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:45 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:08:45.655 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:45 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:08:45.656 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:45 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:45.656 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:45.656 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:45.656 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:45.656 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:45.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:45 smithi117 ceph-mon[32835]: pgmap v1145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:45 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:08:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:45 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:08:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:45 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:45.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:45 smithi107 ceph-mon[32381]: pgmap v1145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:45 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:08:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:45 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:08:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:45 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:08:46.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:46.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:46.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:46.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:46.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:46.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:46.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:46.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:46.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:46.977 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:46.980 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:46.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:47.093 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:47.094 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":387,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:08:41.440061+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:47.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:47.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:47.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:47.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:47.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:47.598 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:08:47.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:47 smithi107 ceph-mon[32381]: pgmap v1146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:47 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:08:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:47 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:08:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/673519404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:47.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:47.795 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:47.795 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:47.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:47 smithi086 ceph-mon[29229]: pgmap v1146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:47.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:47 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:08:47.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:47 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:08:47.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:47.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/673519404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:47 smithi117 ceph-mon[32835]: pgmap v1146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:47 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:08:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:47 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:08:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:08:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/673519404' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:47.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:48.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:48.073 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:48.073 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:48.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:48.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:48.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:48.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:48.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:48.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:48.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:48.731 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:48.731 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:48.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:48 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:48.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:48.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:48 smithi086 ceph-mon[29229]: osdmap e388: 12 total, 3 up, 11 in 2024-08-06T18:08:48.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:48.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:48 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:48.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:48.867 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:48 smithi117 ceph-mon[32835]: osdmap e388: 12 total, 3 up, 11 in 2024-08-06T18:08:48.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:48 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:08:48.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2852551227' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:08:48.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:48 smithi107 ceph-mon[32381]: osdmap e388: 12 total, 3 up, 11 in 2024-08-06T18:08:48.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:49.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:49.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:49.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:49.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:49.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:49.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:49.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:49.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:49.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:49 smithi107 ceph-mon[32381]: pgmap v1148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:49.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:49.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:49 smithi086 ceph-mon[29229]: pgmap v1148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:49.789 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:49.789 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:49 smithi117 ceph-mon[32835]: pgmap v1148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:49.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:50.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:50.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:50.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:50.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:50.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:50.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:50.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:50.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:50.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:50.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:50.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:50.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:50.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:08:50.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:08:50.935 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:08:50.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:08:50.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:51.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:51.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:51.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:51.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:51.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:51.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:51.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:51.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:51.647 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:51.647 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:51.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:51 smithi107 ceph-mon[32381]: pgmap v1149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:51.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2335635450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:51.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.647 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:51 smithi107 ceph-mon[32381]: osdmap e389: 12 total, 3 up, 11 in 2024-08-06T18:08:51.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:51.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:51 smithi086 ceph-mon[29229]: pgmap v1149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:51.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2335635450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:51.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:51 smithi086 ceph-mon[29229]: osdmap e389: 12 total, 3 up, 11 in 2024-08-06T18:08:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:51 smithi117 ceph-mon[32835]: pgmap v1149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2335635450' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:08:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:51 smithi117 ceph-mon[32835]: osdmap e389: 12 total, 3 up, 11 in 2024-08-06T18:08:51.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:51.962 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:08:51.962 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:08:52.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:52.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:52.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:52.176 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:52.176 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:52.222 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:52.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:52.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:52.442 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:52.442 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:52.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:52.610 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:52.610 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:52.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:08:52.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:52.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:52.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:52.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:52.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:52.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:08:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:52.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:52.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:53.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:53.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:53.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:53.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:53.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:53.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:53.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:53.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:53.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:53.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:08:53.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:53.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:53 smithi086 ceph-mon[29229]: pgmap v1151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:53 smithi117 ceph-mon[32835]: pgmap v1151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:53.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:53 smithi107 ceph-mon[32381]: pgmap v1151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:54.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:54.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:54.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:54.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:54.297 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:54.407 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:08:54.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:08:54.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:54.443 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:54.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:54.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:54.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:54.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:54.709 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:54.710 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:54.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:54.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:55.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:55.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:55.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:55.234 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:08:55.234 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":389,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:08:51.442651+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:08:55.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:55.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:55.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:55.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:08:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:55 smithi107 ceph-mon[32381]: pgmap v1152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:55.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1337230381' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:55.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:55 smithi086 ceph-mon[29229]: pgmap v1152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1337230381' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:55 smithi117 ceph-mon[32835]: pgmap v1152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1337230381' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:08:55.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:08:56.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:56.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:56.204 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:08:56.204 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:08:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:08:56.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:08:56.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:08:56.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:56.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:08:56.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:08:56.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:08:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:08:56.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:56.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:56.879 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:56.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:57.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:57.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:57.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:08:57.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:57.620 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:08:57.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:57.681 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:57 smithi086 ceph-mon[29229]: pgmap v1153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:57.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:57.816 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:57 smithi107 ceph-mon[32381]: pgmap v1153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:57 smithi117 ceph-mon[32835]: pgmap v1153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:57.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:58.061 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:58.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:58.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:08:58.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:58.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:58.613 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:08:58.613 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:08:58.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:08:58.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:59.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:59.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:59.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:08:59.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:08:59.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:08:59.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:59.621 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:08:59.621 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:08:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:08:59.623 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:08:59 smithi086 ceph-mon[29229]: pgmap v1154: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:59.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:08:59.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:08:59.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:08:59.837 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:08:59 smithi117 ceph-mon[32835]: pgmap v1154: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:08:59.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:08:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:08:59 smithi107 ceph-mon[32381]: pgmap v1154: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:00.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:00.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:00.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:00.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:00.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:00.647 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:09:00.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:00.854 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:00.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:09:00.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:09:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:09:00.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:00.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:09:00.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:01.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:01.086 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:01.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:01.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:01.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:01.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:01.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:01.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:01.649 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:01.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:01.703 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:01 smithi107 ceph-mon[32381]: pgmap v1155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:01.703 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:01 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:01.704 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:01.704 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:01 smithi107 ceph-mon[32381]: osdmap e390: 12 total, 3 up, 11 in 2024-08-06T18:09:01.704 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:01 smithi107 ceph-mon[32381]: osdmap e391: 12 total, 3 up, 11 in 2024-08-06T18:09:01.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:01.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:01 smithi086 ceph-mon[29229]: pgmap v1155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:01.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:01 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:01 smithi086 ceph-mon[29229]: osdmap e390: 12 total, 3 up, 11 in 2024-08-06T18:09:01.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:01 smithi086 ceph-mon[29229]: osdmap e391: 12 total, 3 up, 11 in 2024-08-06T18:09:01.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:02.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:01 smithi117 ceph-mon[32835]: pgmap v1155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:01 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2802492946' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:01 smithi117 ceph-mon[32835]: osdmap e390: 12 total, 3 up, 11 in 2024-08-06T18:09:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:01 smithi117 ceph-mon[32835]: osdmap e391: 12 total, 3 up, 11 in 2024-08-06T18:09:02.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:02.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:02.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:02.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:02.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:02.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:02.937 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:02.937 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:02.937 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:02.944 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:02.944 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:02.944 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:02.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:02.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:02.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:02.962 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:02.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:02.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:02.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:02.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:02.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:03.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:03.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:03.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:03.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:03.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:03.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:09:03.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:03.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:03 smithi117 ceph-mon[32835]: pgmap v1158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3057180509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:03.900 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:03.900 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:03.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:03 smithi086 ceph-mon[29229]: pgmap v1158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:03.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3057180509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:03.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:03.900 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:03.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:03 smithi107 ceph-mon[32381]: pgmap v1158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3057180509' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:04.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:04.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:04.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:04.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:04.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:04.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:04.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:04.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:04.916 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:04.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:05.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:05.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:05.165 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:05.166 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:05.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:05.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:05.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:05.632 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:05 smithi107 ceph-mon[32381]: pgmap v1159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:05.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:05.705 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:05 smithi086 ceph-mon[29229]: pgmap v1159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:05.795 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:05.795 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":391,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:01.444965+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:05.808 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:09:05.808 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:259,from:0,reservation_nonce:0)", 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 565.03818758299997, 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000226534, 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:09:05.809 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771672+0000", 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.810 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771673+0000", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0399999999999995e-07 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771676+0000", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7480000000000002e-06 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771685+0000", 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1759999999999999e-06 2024-08-06T18:09:05.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771690+0000", 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2950000000000001e-06 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771734+0000", 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4218000000000003e-05 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:09:05.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771750+0000", 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6150999999999998e-05 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.771899+0000", 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014824200000000001 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.813 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 565.037469203, 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8885000000000002e-05, 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.814 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772391+0000", 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0699999999999998e-07 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:09:05.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772394+0000", 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5969999999999999e-06 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772400+0000", 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9539999999999998e-06 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772404+0000", 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6399999999999999e-06 2024-08-06T18:09:05.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772435+0000", 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0793999999999997e-05 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772441+0000", 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.224e-06 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T17:59:40.772459+0000", 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8269000000000001e-05 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:303,from:0,reservation_nonce:0)", 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 398.77194121399998, 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021705499999999999, 2024-08-06T18:09:05.818 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037919+0000", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.6899999999999996e-07 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037923+0000", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2150000000000001e-06 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037932+0000", 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2110000000000004e-06 2024-08-06T18:09:05.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037937+0000", 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6019999999999999e-06 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.037982+0000", 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5108000000000002e-05 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:09:05.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038001+0000", 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8410000000000002e-05 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038136+0000", 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013474 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.822 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 398.77127337799999, 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.1237999999999998e-05, 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038586+0000", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038587+0000", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9999999999999997e-07 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038591+0000", 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6270000000000002e-06 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038599+0000", 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4489999999999999e-06 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038604+0000", 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5190000000000003e-06 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:09:05.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038634+0000", 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0531999999999998e-05 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038640+0000", 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5099999999999998e-06 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:02:27.038658+0000", 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8000999999999999e-05 2024-08-06T18:09:05.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:09:05.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:09:05.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:09:05.827 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:09:05.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:05.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:05 smithi117 ceph-mon[32835]: pgmap v1159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:05.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:05.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:05.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:06.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:06.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:06.277 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:06.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:06.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:06.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:06.532 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:06.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:06.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:06.792 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/164550449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:06.792 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:06.792 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:06.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/164550449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:06.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:06.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/164550449' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:06.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:07.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:07.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:09:07.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:07.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:07.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:07.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:07.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:07.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:07.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:07.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:07.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:07 smithi086 ceph-mon[29229]: pgmap v1160: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:07.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:09:07.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:07 smithi107 ceph-mon[32381]: pgmap v1160: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:07.913 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:07.913 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:07.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:07.997 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:07 smithi117 ceph-mon[32835]: pgmap v1160: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:08.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:08.147 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:08.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:08.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:08.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:08.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:08.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:08.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:08.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:08.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:08.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:08.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:08.997 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:09.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:09.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:09.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:09.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:09.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:09.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:09.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:09.642 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:09:09.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:09.776 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:09.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:09.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:09.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: pgmap v1161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3723315100,config=507150784,health=3595166731,kv=2813548544}) 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1431804412,logm=3307808514}) 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059422122}) 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605335385}) 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1701482516}) 2024-08-06T18:09:09.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2431020896}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587724906}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554063278}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643105422}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964596696}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108615047}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1533933322}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249145958}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926247237}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632124436}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159427532}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984520954}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358196404}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784992749}) 2024-08-06T18:09:09.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267410926}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988221857}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448743671}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1055565006}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017949312}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365564895}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268808904}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635275815}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364441751}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454478153}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623258324}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372753499}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247064427}) 2024-08-06T18:09:09.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960363716}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287707371}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437280670}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2877823930}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516490326}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481490487}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704275590}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513624482}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261893374}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981413566}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343199669}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5777185}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=4157077251,mdsmap=817224545,mgr=2923094491}) 2024-08-06T18:09:09.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=86} crc {mgr=3093430750,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1842625351,mon_config_key=948917129}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2617895042}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=1263101399,monmap=2720365863,osd_metadata=2726447651}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=3205480894,osd_pg_creating=3296361640,osdmap=1255969841}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=610226964}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2774375165}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3398304746}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=1401225589}) 2024-08-06T18:09:09.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:09.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: pgmap v1161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3723315100,config=507150784,health=3595166731,kv=2813548544}) 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1431804412,logm=3307808514}) 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059422122}) 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605335385}) 2024-08-06T18:09:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1701482516}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2431020896}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587724906}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554063278}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643105422}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964596696}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108615047}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1533933322}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249145958}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926247237}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632124436}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159427532}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984520954}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358196404}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784992749}) 2024-08-06T18:09:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267410926}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988221857}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448743671}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1055565006}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017949312}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365564895}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268808904}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635275815}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364441751}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454478153}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623258324}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372753499}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247064427}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960363716}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287707371}) 2024-08-06T18:09:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437280670}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2877823930}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516490326}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481490487}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704275590}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513624482}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261893374}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981413566}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343199669}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5777185}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=4157077251,mdsmap=817224545,mgr=2923094491}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=86} crc {mgr=3093430750,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1842625351,mon_config_key=948917129}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2617895042}) 2024-08-06T18:09:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=1263101399,monmap=2720365863,osd_metadata=2726447651}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=3205480894,osd_pg_creating=3296361640,osdmap=1255969841}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=610226964}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2774375165}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3398304746}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=1401225589}) 2024-08-06T18:09:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: pgmap v1161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=32,config=2,health=12,kv=54} crc {auth=3723315100,config=507150784,health=3595166731,kv=2813548544}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {kv=4,logm=96} crc {kv=1431804412,logm=3307808514}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3059422122}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2605335385}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1701482516}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2431020896}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3587724906}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1554063278}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=643105422}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1964596696}) 2024-08-06T18:09:09.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3108615047}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1533933322}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2249145958}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=926247237}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632124436}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1159427532}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3984520954}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2358196404}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1784992749}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1267410926}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=988221857}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448743671}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1055565006}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3017949312}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365564895}) 2024-08-06T18:09:09.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268808904}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1635275815}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2364441751}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454478153}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2623258324}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=372753499}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2247064427}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=960363716}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4287707371}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1437280670}) 2024-08-06T18:09:09.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2877823930}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=516490326}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2481490487}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=704275590}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3513624482}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=261893374}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1981413566}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2343199669}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=5777185}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=80,mdsmap=3,mgr=17} crc {logm=4157077251,mdsmap=817224545,mgr=2923094491}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mgr=1,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=86} crc {mgr=3093430750,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=1842625351,mon_config_key=948917129}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2617895042}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=93,monmap=5,osd_metadata=2} crc {mon_config_key=1263101399,monmap=2720365863,osd_metadata=2726447651}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osd_metadata=10,osd_pg_creating=1,osdmap=89} crc {osd_metadata=3205480894,osd_pg_creating=3296361640,osdmap=1255969841}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=610226964}) 2024-08-06T18:09:09.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2774375165}) 2024-08-06T18:09:09.980 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3398304746}) 2024-08-06T18:09:09.980 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:09:09.980 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=63} crc {osdmap=1401225589}) 2024-08-06T18:09:09.980 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:10.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:10.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:10.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:10.431 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:10.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:09:10.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:10.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:10.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:10.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:10.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:10.655 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:10.655 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:10.683 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:10.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:10.861 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:10.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:10.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:10 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:10.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:10.911 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:10 smithi117 ceph-mon[32835]: osdmap e392: 12 total, 3 up, 11 in 2024-08-06T18:09:10.948 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:10.948 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:10.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:10 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:10.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:10.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:10 smithi086 ceph-mon[29229]: osdmap e392: 12 total, 3 up, 11 in 2024-08-06T18:09:10.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:10 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/290041183' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:10 smithi107 ceph-mon[32381]: osdmap e392: 12 total, 3 up, 11 in 2024-08-06T18:09:11.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:11.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:11.212 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:11.212 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:11.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:11.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:11.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:11.314 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:11.314 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:11.331 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:11.332 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:11.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:11.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:11.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:11.348 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:11.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:11.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:11.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:11.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:11.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:11.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:11.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:11 smithi107 ceph-mon[32381]: pgmap v1163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:11.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4019980246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:11.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:11 smithi107 ceph-mon[32381]: osdmap e393: 12 total, 3 up, 11 in 2024-08-06T18:09:11.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:11.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:11 smithi086 ceph-mon[29229]: pgmap v1163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:11.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4019980246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:11.834 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:11 smithi086 ceph-mon[29229]: osdmap e393: 12 total, 3 up, 11 in 2024-08-06T18:09:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:11 smithi117 ceph-mon[32835]: pgmap v1163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4019980246' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:11 smithi117 ceph-mon[32835]: osdmap e393: 12 total, 3 up, 11 in 2024-08-06T18:09:11.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:12.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:12.098 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:12.098 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:12.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:12.322 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:12.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:09:12.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:12.351 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:12.351 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:12.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:12.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:12.720 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:12.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:12.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:12.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:12.871 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:12.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:12.973 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:12.973 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:12.973 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:13.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:13.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:13.160 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:13.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:09:13.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:13.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:13.409 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:13.443 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:13.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:13.554 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:13.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:13.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:13.697 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:13 smithi117 ceph-mon[32835]: pgmap v1165: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:13.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:13.828 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:13 smithi107 ceph-mon[32381]: pgmap v1165: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:13.832 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:13.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:13 smithi086 ceph-mon[29229]: pgmap v1165: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:13.832 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":393,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:11.447884+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:13.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:14.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:14.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:14.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:14.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:14.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:14.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:14.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:14.632 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:14.632 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:14.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:09:14.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:14.765 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1654132353' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:14.864 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1654132353' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:14.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:14.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1654132353' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:14.970 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:15.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:15.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:15.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:15.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:15.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:15.559 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:15.559 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:15.599 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:15.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:09:15.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:15.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:15.826 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:15 smithi107 ceph-mon[32381]: pgmap v1166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:15.854 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:15 smithi086 ceph-mon[29229]: pgmap v1166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:15.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:15.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:15.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:16.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:16.086 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:16.087 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:15 smithi117 ceph-mon[32835]: pgmap v1166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:16.126 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:16.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:16.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:16.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:16.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:16.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:16.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:16.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:16.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:09:16.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:16.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:16.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:16.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:17.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:17.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:17.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:17.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:17.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:17.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:17.457 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:17.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:17.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:17.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:17.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:17.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:17 smithi107 ceph-mon[32381]: pgmap v1167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:17.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:17 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:17.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:17.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:17 smithi107 ceph-mon[32381]: osdmap e394: 12 total, 3 up, 11 in 2024-08-06T18:09:17.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:17.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:17 smithi086 ceph-mon[29229]: pgmap v1167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:17 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:17 smithi086 ceph-mon[29229]: osdmap e394: 12 total, 3 up, 11 in 2024-08-06T18:09:17.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:17.967 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:17 smithi117 ceph-mon[32835]: pgmap v1167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:17.967 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:17 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:17.967 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1961553519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:17.968 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:17 smithi117 ceph-mon[32835]: osdmap e394: 12 total, 3 up, 11 in 2024-08-06T18:09:18.074 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:18.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:18.172 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:18.172 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:18.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:09:18.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:18.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:18.455 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:18.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:18.551 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:18.552 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:18.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:18.568 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:18.569 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:18.569 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:18.569 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:18.569 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:18.586 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:18.586 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:18.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:18.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:18.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:18.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:18.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:18.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:18.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:18.759 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3464908108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:18.759 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:18.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3464908108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:18.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:18.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:18.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3464908108' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:18.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:19.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:19.014 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:19.014 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:19.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:19.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:19.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:19.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:09:19.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:19.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:19.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:19.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:19 smithi086 ceph-mon[29229]: pgmap v1169: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:19.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:19 smithi107 ceph-mon[32381]: pgmap v1169: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:19.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:19.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:19.921 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:19.921 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:19.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:20.000 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:20.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:20.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:19 smithi117 ceph-mon[32835]: pgmap v1169: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:20.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:20.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:20.132 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:20.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:20.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:20.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:20.364 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:20.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:20.495 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:20.496 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:20.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:20.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:20.665 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:20.738 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:20.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:09:20.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:20.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:20.947 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:20.948 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:20.985 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:21.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:21.082 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:21.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:21.206 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:21.206 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:21.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:21.244 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":394,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:16.673797+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:20.226924+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:21.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:21.447 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:21.447 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:21.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:21.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:21.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:21 smithi107 ceph-mon[32381]: pgmap v1170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2732277835' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:21 smithi107 ceph-mon[32381]: osdmap e395: 12 total, 3 up, 11 in 2024-08-06T18:09:21.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:21.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:21 smithi086 ceph-mon[29229]: pgmap v1170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2732277835' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:21 smithi086 ceph-mon[29229]: osdmap e395: 12 total, 3 up, 11 in 2024-08-06T18:09:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:21.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:22.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:09:22.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:21 smithi117 ceph-mon[32835]: pgmap v1170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:22.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2732277835' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:22.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:21 smithi117 ceph-mon[32835]: osdmap e395: 12 total, 3 up, 11 in 2024-08-06T18:09:22.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:22.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:22.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:22.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:22.614 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:22.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:22.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:22.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:23.000 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:23.000 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:23.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:23.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:23.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:23.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:09:23.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:23.354 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:23.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:23.558 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:23.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:23.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:23.713 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:09:23.761 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:23 smithi107 ceph-mon[32381]: pgmap v1172: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:23.761 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:23.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:23.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:23 smithi086 ceph-mon[29229]: pgmap v1172: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:23.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:23.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:23.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:23.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:24.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:23 smithi117 ceph-mon[32835]: pgmap v1172: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:24.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:24.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:24.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:24.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:24.503 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:24.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:24.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:24 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:24 smithi117 ceph-mon[32835]: osdmap e396: 12 total, 3 up, 11 in 2024-08-06T18:09:24.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:24 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:24 smithi107 ceph-mon[32381]: osdmap e396: 12 total, 3 up, 11 in 2024-08-06T18:09:25.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:24 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2761776162' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:24 smithi086 ceph-mon[29229]: osdmap e396: 12 total, 3 up, 11 in 2024-08-06T18:09:25.170 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:25.209 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:09:25.209 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:09:25.209 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:09:25.209 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:09:25.306 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:25.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:25.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:25.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:25.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:25.703 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:25.704 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:25.704 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:25.704 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:25.704 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:25.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:25.716 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:25 smithi086 ceph-mon[29229]: pgmap v1174: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:25.716 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1772795285' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:25.716 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:25.716 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:25.722 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:25.723 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:25.723 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:25.724 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:25.724 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:25.724 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:25.724 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:25.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:25.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:25 smithi107 ceph-mon[32381]: pgmap v1174: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:25.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1772795285' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:25.974 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:25.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:09:25.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:25 smithi117 ceph-mon[32835]: pgmap v1174: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:25.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1772795285' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:26.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:26.230 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:26.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:26.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:26.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:26.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:26.515 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:26.515 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:26.591 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:26.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:26.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:26.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:26.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:26.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:09:26.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:09:26.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:26 smithi107 ceph-mon[32381]: osdmap e397: 12 total, 3 up, 11 in 2024-08-06T18:09:26.980 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:26.980 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:26.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:26.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:09:26.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:09:26.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:26 smithi086 ceph-mon[29229]: osdmap e397: 12 total, 3 up, 11 in 2024-08-06T18:09:27.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:27.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:27.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:27.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:27.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:09:27.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:09:27.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:26 smithi117 ceph-mon[32835]: osdmap e397: 12 total, 3 up, 11 in 2024-08-06T18:09:27.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:27.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:27.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:27.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:27.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:27.814 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:09:27.814 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:09:27.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:27.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:27 smithi117 ceph-mon[32835]: pgmap v1175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:27.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:27 smithi107 ceph-mon[32381]: pgmap v1175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:27.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:27 smithi086 ceph-mon[29229]: pgmap v1175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:27.998 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:27.999 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:28.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:28.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:28.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:28.400 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:28.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:09:28.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:28.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:09:28.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:28.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:28.570 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:28.570 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":397,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:26.451978+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:28.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:28.678 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:28.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:28.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4097944075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:28.857 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:28.858 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:28.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:28.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4097944075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:29.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:29.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:29.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4097944075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:29.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:29.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:29.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:29.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:29.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:29.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:29.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:29 smithi086 ceph-mon[29229]: pgmap v1177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:29.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:29.983 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:29 smithi107 ceph-mon[32381]: pgmap v1177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:30.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:30.103 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:29 smithi117 ceph-mon[32835]: pgmap v1177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:30.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:30.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:30.352 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:09:30.353 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:09:30.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:30.499 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:30.500 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:30.534 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:30.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:30.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:30.760 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:30 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:09:30.761 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:30 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:09:30.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:30 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:09:30.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:30 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:09:30.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:31.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:31.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:09:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:30 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:09:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:30 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:09:31.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:31.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:31.308 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:31.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:31.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:31.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:31.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:31 smithi086 ceph-mon[29229]: pgmap v1178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:31.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:31 smithi107 ceph-mon[32381]: pgmap v1178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:32.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:32.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:31 smithi117 ceph-mon[32835]: pgmap v1178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:32.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:32.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:32.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:32.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:32.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:32.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:32.679 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:32.679 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:32.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:32.697 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:32.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:32.698 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:32.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:32.742 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:09:32.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:32.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2993296018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:32.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:32.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:32.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:32.920 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:33.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:33.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:33.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2993296018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:33.024 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:33.024 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:09:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2993296018' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:33.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:33.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:33.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:33.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:33.649 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:33.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:09:33.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:33.745 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:33.745 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:33.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: pgmap v1179: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:33 smithi117 ceph-mon[32835]: osdmap e398: 12 total, 3 up, 11 in 2024-08-06T18:09:33.900 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:33.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: pgmap v1179: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:33.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:33 smithi107 ceph-mon[32381]: osdmap e398: 12 total, 3 up, 11 in 2024-08-06T18:09:34.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: pgmap v1179: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/618236862' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:09:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:33 smithi086 ceph-mon[29229]: osdmap e398: 12 total, 3 up, 11 in 2024-08-06T18:09:34.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:34.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:34.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:34.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:34.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:34.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:34.734 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:34.734 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:34.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:34.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:35.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:35.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:35.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:35.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:35.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:35.411 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:35.411 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:35.437 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:35.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:35.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:35.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:09:35.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:35.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:35 smithi117 ceph-mon[32835]: pgmap v1181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:35.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:35.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:35 smithi107 ceph-mon[32381]: pgmap v1181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:36.005 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:36.019 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:36.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:09:36.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:35 smithi086 ceph-mon[29229]: pgmap v1181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:36.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:36.131 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":398,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:32.737791+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:08.131626+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:36.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:09:36.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:36.308 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:09:36.325 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:36.345 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:36.346 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:36.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:36.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:36.661 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:36.661 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:36.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:36.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:36.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:36 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:09:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:36 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:09:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2229820489' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:36.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:36 smithi107 ceph-mon[32381]: osdmap e399: 12 total, 3 up, 11 in 2024-08-06T18:09:36.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:37.015 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:37.015 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:37.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:36 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:09:37.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:36 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:09:37.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2229820489' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:37.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:36 smithi086 ceph-mon[29229]: osdmap e399: 12 total, 3 up, 11 in 2024-08-06T18:09:37.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:36 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:09:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:36 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:09:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2229820489' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:36 smithi117 ceph-mon[32835]: osdmap e399: 12 total, 3 up, 11 in 2024-08-06T18:09:37.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:37.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:37.456 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:37.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:37.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:37.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:37.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:37 smithi117 ceph-mon[32835]: pgmap v1182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:37.893 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:37.893 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:37 smithi107 ceph-mon[32381]: pgmap v1182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:37.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:37.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:37.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:38.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:38.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:37 smithi086 ceph-mon[29229]: pgmap v1182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:38.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:38.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:38.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:38.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:38.341 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:38.341 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:38.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:38.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:09:38.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:38.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:38.746 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:38.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:38.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:39.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:39.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:39.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:39.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:39.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:39.637 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:39.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:39.654 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:39.654 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:39.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:39.671 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:39.671 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:39.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:39.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:39.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:39 smithi086 ceph-mon[29229]: pgmap v1184: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:39.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3063097132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:39.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:39.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:39.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:39 smithi107 ceph-mon[32381]: pgmap v1184: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:39.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3063097132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:39.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:40.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:39 smithi117 ceph-mon[32835]: pgmap v1184: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:40.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3063097132' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:40.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:40.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:40.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:40.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:40.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:40.391 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:40.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:40.579 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:40.579 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:40.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:40.608 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:40.701 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:40.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:40.756 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:09:40.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:40.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:40.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:40.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:40.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:41.013 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:41.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:09:41.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:41.141 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:41.141 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:41.141 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:41.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:41.271 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:41.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:41.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:41.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:41.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:41.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:41.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:41 smithi086 ceph-mon[29229]: pgmap v1185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:41 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:41 smithi086 ceph-mon[29229]: osdmap e400: 12 total, 3 up, 11 in 2024-08-06T18:09:41.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:41 smithi086 ceph-mon[29229]: osdmap e401: 12 total, 3 up, 11 in 2024-08-06T18:09:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:41 smithi117 ceph-mon[32835]: pgmap v1185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:41 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:41.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:41 smithi117 ceph-mon[32835]: osdmap e400: 12 total, 3 up, 11 in 2024-08-06T18:09:41.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:41 smithi117 ceph-mon[32835]: osdmap e401: 12 total, 3 up, 11 in 2024-08-06T18:09:41.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:42.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:42.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:42.197 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:42.197 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:41 smithi107 ceph-mon[32381]: pgmap v1185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:41 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/298035747' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:41 smithi107 ceph-mon[32381]: osdmap e400: 12 total, 3 up, 11 in 2024-08-06T18:09:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:41 smithi107 ceph-mon[32381]: osdmap e401: 12 total, 3 up, 11 in 2024-08-06T18:09:42.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:42.440 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:42.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:42.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:09:42.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:42.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:42.939 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:42.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:43.103 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:43.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:43.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:43.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:43.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:43.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:43.230 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:43.230 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":401,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:41.456505+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:08:29.151066+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:43.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:43.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:43.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:43.527 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:43.527 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:43.613 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:43.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:09:43.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:43.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:43.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:43 smithi086 ceph-mon[29229]: pgmap v1188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:43.789 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/989102226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:43.789 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:43.789 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:43.861 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:43 smithi117 ceph-mon[32835]: pgmap v1188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/989102226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:43.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:43.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:09:44.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:44.059 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:43 smithi107 ceph-mon[32381]: pgmap v1188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:44.059 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/989102226' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:44.059 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:44.059 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:44.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:44.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:44.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:44.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:44.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:44.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:44.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:44.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:44.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:45.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:45.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:45.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:45.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:45.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:45.469 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:45.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:45.541 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:45.541 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:45.541 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:45.543 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:45.543 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:45.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:45.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:45.544 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:45.561 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:45.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:45.563 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:45.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:45.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:45.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:45.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:45.895 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:45.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:45.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:45 smithi107 ceph-mon[32381]: pgmap v1189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1950980416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:46.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:46.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:45 smithi117 ceph-mon[32835]: pgmap v1189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:46.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1950980416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:46.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:45 smithi086 ceph-mon[29229]: pgmap v1189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:46.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1950980416' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:46.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:46.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:09:46.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:46.229 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:46.230 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:46.275 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:46.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:46.392 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:46.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:46.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:46.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:46.765 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:46.765 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:46.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:46.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:47.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:47.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:47.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:47.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:47.390 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:47.390 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:47.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:47.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:47.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:47.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:47.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:47.941 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:48.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:48.064 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:47 smithi086 ceph-mon[29229]: pgmap v1190: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:48.064 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:48.065 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:48.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:47 smithi107 ceph-mon[32381]: pgmap v1190: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:48.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:48.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:47 smithi117 ceph-mon[32835]: pgmap v1190: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:48.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:48.199 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:48.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:09:48.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:48.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:48.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:48.604 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:48.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:09:48.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:48.773 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:09:48.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:48.848 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:48.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:48.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:49.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:49.122 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:09:49.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:49.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:49.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:49.610 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:49.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:49.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:49.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:49 smithi086 ceph-mon[29229]: pgmap v1191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:49.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:49 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:49.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:49.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:49 smithi086 ceph-mon[29229]: osdmap e402: 12 total, 3 up, 11 in 2024-08-06T18:09:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:49 smithi117 ceph-mon[32835]: pgmap v1191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:49 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:49 smithi117 ceph-mon[32835]: osdmap e402: 12 total, 3 up, 11 in 2024-08-06T18:09:49.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:50.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:50.043 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:49 smithi107 ceph-mon[32381]: pgmap v1191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:50.043 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:49 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:09:50.043 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/223354626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:09:50.043 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:49 smithi107 ceph-mon[32381]: osdmap e402: 12 total, 3 up, 11 in 2024-08-06T18:09:50.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:50.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:50.328 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:50.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:50.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:50.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:50.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:50.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:50.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:50.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:51.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:51.087 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:51.088 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:09:51.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:51.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:51.386 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:51.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:51.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:09:51.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:51.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:51.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:51.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:51.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:51 smithi086 ceph-mon[29229]: pgmap v1193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:51.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:51 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:09:51.800 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:51 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:09:51.800 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:51 smithi086 ceph-mon[29229]: osdmap e403: 12 total, 3 up, 11 in 2024-08-06T18:09:51.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:52.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:52.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:52.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:52.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:51 smithi107 ceph-mon[32381]: pgmap v1193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:52.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:51 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:09:52.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:51 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:09:52.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:51 smithi107 ceph-mon[32381]: osdmap e403: 12 total, 3 up, 11 in 2024-08-06T18:09:52.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:52.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:51 smithi117 ceph-mon[32835]: pgmap v1193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:51 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:09:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:51 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:09:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:51 smithi117 ceph-mon[32835]: osdmap e403: 12 total, 3 up, 11 in 2024-08-06T18:09:52.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:52.216 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:52.217 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:52.217 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:52.217 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:52.217 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:52.235 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:52.236 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:52.237 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:09:52.287 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:52.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:52.425 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:52.425 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":403,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:51.459181+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:52.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:52.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:52.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:52.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:52.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:52.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:52.943 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:53.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:53.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/317161087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:53.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2699910970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:53.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:53.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/317161087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2699910970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:53.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:53.210 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/317161087' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:09:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2699910970' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:09:53.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:53.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:53.502 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:53.502 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:53.576 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:09:53.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:09:53.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:53.614 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:53.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:09:53.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:53 smithi086 ceph-mon[29229]: pgmap v1195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:53 smithi117 ceph-mon[32835]: pgmap v1195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:53.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:53.925 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:54.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:54.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:53 smithi107 ceph-mon[32381]: pgmap v1195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:09:54.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:54.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:54.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:54.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:54.703 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:54.703 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:54.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:54.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:54 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:09:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:54 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:09:54.970 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:54 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:09:54.970 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:54 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:09:55.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:55.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:55.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:55.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:55.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:54 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:09:55.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:54 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:09:55.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:55.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:55.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:55.458 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:55.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:55.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:09:55.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:55.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:55.796 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:09:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:55 smithi086 ceph-mon[29229]: pgmap v1196: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:55.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:55.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:56.009 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:55 smithi117 ceph-mon[32835]: pgmap v1196: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:56.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:56.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:09:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:55 smithi107 ceph-mon[32381]: pgmap v1196: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:56.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:09:56.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:09:56.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:56.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:56.315 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:56.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:09:56.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:56.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:09:56.562 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:56.564 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:56.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:56.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:56.812 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:56 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:56.812 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:56.812 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:56 smithi107 ceph-mon[32381]: osdmap e404: 12 total, 3 up, 11 in 2024-08-06T18:09:56.812 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:56 smithi107 ceph-mon[32381]: osdmap e405: 12 total, 3 up, 11 in 2024-08-06T18:09:56.914 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:56 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:56.914 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:56.914 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:56 smithi086 ceph-mon[29229]: osdmap e404: 12 total, 3 up, 11 in 2024-08-06T18:09:56.914 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:56 smithi086 ceph-mon[29229]: osdmap e405: 12 total, 3 up, 11 in 2024-08-06T18:09:56.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:56.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:56 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:09:56.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2077106671' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:09:56.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:56 smithi117 ceph-mon[32835]: osdmap e404: 12 total, 3 up, 11 in 2024-08-06T18:09:56.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:56 smithi117 ceph-mon[32835]: osdmap e405: 12 total, 3 up, 11 in 2024-08-06T18:09:57.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:57.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:57.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:09:57.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:57.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:57.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:57.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:57.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:57.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:09:57.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:09:57.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:57.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:09:57.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:57.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:57 smithi117 ceph-mon[32835]: pgmap v1198: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:57.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:57 smithi086 ceph-mon[29229]: pgmap v1198: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:57.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:58.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:58.135 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:57 smithi107 ceph-mon[32381]: pgmap v1198: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:58.189 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:09:58.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:58.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:58.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:58.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:58.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:09:58.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:09:58.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:58.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:58.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:58.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:58.835 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:09:58.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:09:58.873 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:09:58.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:59.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:09:59.100 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:59.101 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:59.118 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:09:59.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:09:59.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:09:59.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:09:59.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:09:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:09:59.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:09:59.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:59.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:09:59.821 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:09:59.822 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":405,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:09:56.460862+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:09:59.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:09:59.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:59 smithi086 ceph-mon[29229]: pgmap v1200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:09:59.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:09:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:09:59.936 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:09:59.937 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:09:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:09:59.945 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:09:59.946 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:09:59.946 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:09:59.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:09:59.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:09:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:09:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:09:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:09:59.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:09:59.981 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:09:59.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:09:59.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:09:59.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:09:59.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:09:59.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:09:59.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:00.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:59 smithi117 ceph-mon[32835]: pgmap v1200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:09:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:10:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:59 smithi107 ceph-mon[32381]: pgmap v1200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:09:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:10:00.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:00.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:09:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:00.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:00.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:00.259 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:00.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:00.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:00.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:00.722 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:00.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:00.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:01.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3256645170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3007516278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: Health detail: HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:10:01.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 1.0 is stuck stale for 19m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.2 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck inactive for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.6 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.8 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.9 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.a is stuck stale for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.c is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.d is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.e is stuck inactive for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.f is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.10 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.13 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.15 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.18 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck undersized for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.a is stuck undersized for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.c is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.e is stuck undersized for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.064 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.065 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.065 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.065 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck undersized for 18m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.065 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3256645170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:01.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3007516278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:01.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: Health detail: HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:10:01.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 1.0 is stuck stale for 19m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.2 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck inactive for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.6 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.8 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.9 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.a is stuck stale for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.c is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.d is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.e is stuck inactive for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.f is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.10 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.13 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.15 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.18 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck undersized for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.a is stuck undersized for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.108 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.c is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.e is stuck undersized for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck undersized for 18m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3256645170' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3007516278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: Health detail: HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:10:01.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 1.0 is stuck stale for 19m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.2 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck inactive for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.6 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.8 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.9 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.a is stuck stale for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.c is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.d is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.e is stuck inactive for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.f is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.10 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.13 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.15 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.18 is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck inactive for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.196 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1b is stuck inactive for 19m, current state unknown, last acting [] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck stale for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck inactive for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck undersized for 19m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.a is stuck undersized for 18m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.c is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.e is stuck undersized for 18m, current state undersized+peered, last acting [2] 2024-08-06T18:10:01.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck undersized for 19m, current state undersized+peered, last acting [6] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck undersized for 19m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck undersized for 18m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck undersized for 19m, current state undersized+peered, last acting [0] 2024-08-06T18:10:01.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:01.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:01.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:01.453 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:01.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:10:01.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:01.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:01.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:01.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:01 smithi117 ceph-mon[32835]: pgmap v1201: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:01.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:01.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:01 smithi086 ceph-mon[29229]: pgmap v1201: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:02.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:02.125 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:10:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:01 smithi107 ceph-mon[32381]: pgmap v1201: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:02.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:02.234 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:02.234 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:02.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:02.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:02.568 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:02.569 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:02.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:02.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:02.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:03.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:03.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:03.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:03.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:10:03.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:03.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:03.545 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:03.645 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:03.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:03.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:03.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:03.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:03.812 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:10:03.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:03.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:04.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:04.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:03 smithi107 ceph-mon[32381]: pgmap v1202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:04.106 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:03 smithi086 ceph-mon[29229]: pgmap v1202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:04.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:04.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:03 smithi117 ceph-mon[32835]: pgmap v1202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:04.176 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:04.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:10:04.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:04.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:04.432 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:10:04.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:04.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:04.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:04.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:04 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:04 smithi086 ceph-mon[29229]: osdmap e406: 12 total, 3 up, 11 in 2024-08-06T18:10:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:04 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:04.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:04 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:04 smithi107 ceph-mon[32381]: osdmap e406: 12 total, 3 up, 11 in 2024-08-06T18:10:04.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:04.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:04 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:05.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:04 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4230951070' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:04 smithi117 ceph-mon[32835]: osdmap e406: 12 total, 3 up, 11 in 2024-08-06T18:10:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:04 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:05.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:05.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:05.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:05.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:05.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:05.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:05.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:05.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:05.637 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:05.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:05.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:05.900 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:05.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:05 smithi086 ceph-mon[29229]: pgmap v1204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:05.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:10:05.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:10:05.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:05.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:05.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:05.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:05 smithi107 ceph-mon[32381]: pgmap v1204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:10:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:10:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:06.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:06.091 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:05 smithi117 ceph-mon[32835]: pgmap v1204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:06.091 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:10:06.091 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:10:06.092 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:10:06.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:06.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:06.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:06.482 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:06.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:10:06.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:06.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:06.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:06.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:06.779 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:10:06.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:06.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:06.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:06.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:07.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:10:07.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:07.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:07.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:07.489 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:07.489 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:07.489 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:07.506 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:07.506 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:07.507 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:07.524 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:07.524 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:07.524 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:07 smithi086 ceph-mon[29229]: pgmap v1205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:07.524 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:07 smithi086 ceph-mon[29229]: osdmap e407: 12 total, 3 up, 11 in 2024-08-06T18:10:07.525 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:07.525 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:07.525 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:07.525 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:07.525 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:07.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:07.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:07.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:07 smithi107 ceph-mon[32381]: pgmap v1205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:07.692 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:07 smithi107 ceph-mon[32381]: osdmap e407: 12 total, 3 up, 11 in 2024-08-06T18:10:07.785 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:07 smithi117 ceph-mon[32835]: pgmap v1205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:07.785 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:07 smithi117 ceph-mon[32835]: osdmap e407: 12 total, 3 up, 11 in 2024-08-06T18:10:07.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:07.869 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:07.869 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":407,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:06.463848+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:07.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:07.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:07.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:08.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:08.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:08.262 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:08.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:08.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:08.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:08.623 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2526401945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:08.624 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.624 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.624 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.624 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/498831492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2526401945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/498831492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:08.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:08.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2526401945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:08.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.752 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:08.753 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/498831492' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:08.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:08.834 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:08.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:10:08.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:09.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:09.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:09.241 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:09.245 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:09.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:09.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:09.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:09.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:09.684 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:09 smithi086 ceph-mon[29229]: pgmap v1207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:09.684 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:09.685 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:09.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:09 smithi117 ceph-mon[32835]: pgmap v1207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:09.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:09 smithi107 ceph-mon[32381]: pgmap v1207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:10.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:10.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:10.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:10.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:10.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:10.471 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:10.471 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:10.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:10.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:10.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:10.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:10.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:10:10.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:10.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:11.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:11.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:11.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:11.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:11.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:11.505 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:11 smithi086 ceph-mon[29229]: pgmap v1208: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:11.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:11.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:11.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:11 smithi107 ceph-mon[32381]: pgmap v1208: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:11.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:11.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:11 smithi117 ceph-mon[32835]: pgmap v1208: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 321 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:11.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:12.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:12.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:12.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:12.248 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:12.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:12.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:12.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:12.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:12.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:12.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:12.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:13.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:13.122 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:13.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:10:13.141 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:13.179 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:13.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:13.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:13.482 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:10:13.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:13.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:13.668 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:13 smithi107 ceph-mon[32381]: pgmap v1209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:13.669 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:13.669 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:13.669 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:13.669 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:13.669 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:13.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:13.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:13.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:13.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:13 smithi086 ceph-mon[29229]: pgmap v1209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:13.788 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:13 smithi117 ceph-mon[32835]: pgmap v1209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:13.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:14.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:14.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:14.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:14.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:14.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:14.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:14.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:14.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:14.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:14.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:14.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:14 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:14.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:14.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:14 smithi107 ceph-mon[32381]: osdmap e408: 12 total, 3 up, 11 in 2024-08-06T18:10:14.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:14.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:14 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:14.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:14.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:14 smithi086 ceph-mon[29229]: osdmap e408: 12 total, 3 up, 11 in 2024-08-06T18:10:14.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:14 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:14.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/780735114' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:14.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:14 smithi117 ceph-mon[32835]: osdmap e408: 12 total, 3 up, 11 in 2024-08-06T18:10:14.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:15.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:15.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:15.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:15.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:15.521 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:15.521 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:15.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:15 smithi086 ceph-mon[29229]: pgmap v1211: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:15.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:15.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:15.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:15 smithi107 ceph-mon[32381]: pgmap v1211: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:15.798 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:15.798 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:15 smithi117 ceph-mon[32835]: pgmap v1211: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:15.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:16.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:16.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:16.203 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:16.203 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":408,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:13.476940+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:16.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:10:16.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:16.440 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:16.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:16.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:16.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3874617166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:16.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:16 smithi107 ceph-mon[32381]: osdmap e409: 12 total, 3 up, 11 in 2024-08-06T18:10:16.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:16.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:16.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:16.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3874617166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:16.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:16 smithi086 ceph-mon[29229]: osdmap e409: 12 total, 3 up, 11 in 2024-08-06T18:10:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3874617166' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:16 smithi117 ceph-mon[32835]: osdmap e409: 12 total, 3 up, 11 in 2024-08-06T18:10:16.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:17.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:17.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:17.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:17.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:17.254 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:17.269 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:17.269 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:17.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:17.292 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:17.293 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:17.311 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:17.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:17.311 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:17.311 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:17.312 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:17.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:17.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:17.527 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:17.528 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:17.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:17.680 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:17 smithi117 ceph-mon[32835]: pgmap v1212: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:17.680 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2087943776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:17.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:17.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:17 smithi086 ceph-mon[29229]: pgmap v1212: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:17.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2087943776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:17.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:17 smithi107 ceph-mon[32381]: pgmap v1212: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2087943776' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:18.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:18.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:18.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:18.368 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:18.368 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:18.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:18.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:18.623 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.625 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.625 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.671 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:18.671 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:18.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:18.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:18.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:18.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:18.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:19.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:19.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:19.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:19.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:19.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:19.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:19.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:19.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:19.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:19 smithi086 ceph-mon[29229]: pgmap v1214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:19.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:19 smithi107 ceph-mon[32381]: pgmap v1214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:19.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:19 smithi117 ceph-mon[32835]: pgmap v1214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:19.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:20.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:20.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:20.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:20.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:20.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:20.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:10:20.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:20.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:20.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:20.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:20.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:20.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:20.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:20.964 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:20.964 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:21.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:21.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:21.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:21.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:21.525 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:21.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:21.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:21 smithi107 ceph-mon[32381]: pgmap v1215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:10:21.795 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:21.795 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:21.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:21 smithi086 ceph-mon[29229]: pgmap v1215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:21.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:10:21.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:21 smithi117 ceph-mon[32835]: pgmap v1215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:10:21.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:22.001 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:22.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:22.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:22.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:22.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:22.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:22.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:22.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:22.469 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:10:22.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:22.610 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:22.610 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:22.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:22 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:22.611 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:10:22.611 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:22 smithi086 ceph-mon[29229]: osdmap e410: 12 total, 3 up, 11 in 2024-08-06T18:10:22.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:22 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:10:22.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:22 smithi107 ceph-mon[32381]: osdmap e410: 12 total, 3 up, 11 in 2024-08-06T18:10:22.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:22.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:22.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:22 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3591417602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:10:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:22 smithi117 ceph-mon[32835]: osdmap e410: 12 total, 3 up, 11 in 2024-08-06T18:10:22.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:23.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:23.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:23.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:23.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:23.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:23.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:23.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:23 smithi107 ceph-mon[32381]: pgmap v1216: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:23.800 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:23 smithi086 ceph-mon[29229]: pgmap v1216: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:23.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:23 smithi117 ceph-mon[32835]: pgmap v1216: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:23.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:24.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:24.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:10:24.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:24.154 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:24.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:24.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:24.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:24.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:24.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:24.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:24.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:24.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:24.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:24.913 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:24.913 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":410,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:22.469351+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:25.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:25.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:25.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:10:25.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:25.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:25.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:25.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:25.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:25 smithi086 ceph-mon[29229]: pgmap v1218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:25.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2057484808' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:25.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:25 smithi117 ceph-mon[32835]: pgmap v1218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2057484808' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:25.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:25 smithi107 ceph-mon[32381]: pgmap v1218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:25.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2057484808' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:25.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:26.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:26.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:26.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:26.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:26.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:26.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:26.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:26.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:26.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:26.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:26.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:26.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:26.949 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:26.949 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:26.974 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:26.984 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:26.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:27.006 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:27.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:27.024 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:27.024 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:27.025 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:27.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:27.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:27.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:27.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:27.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:27.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:27.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:27 smithi086 ceph-mon[29229]: pgmap v1219: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:27.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:27 smithi086 ceph-mon[29229]: osdmap e411: 12 total, 3 up, 11 in 2024-08-06T18:10:27.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2279358991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:27.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:27 smithi117 ceph-mon[32835]: pgmap v1219: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:27 smithi117 ceph-mon[32835]: osdmap e411: 12 total, 3 up, 11 in 2024-08-06T18:10:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2279358991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:27 smithi107 ceph-mon[32381]: pgmap v1219: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:27 smithi107 ceph-mon[32381]: osdmap e411: 12 total, 3 up, 11 in 2024-08-06T18:10:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2279358991' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:27.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:28.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:28.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:28.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:28.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:28.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:28.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:28.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:28.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:28.687 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:28.688 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:28.688 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:28.691 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:28.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:28.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:28.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:29.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:29.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:29.244 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:29.244 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:29.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:29.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:29.536 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:29 smithi086 ceph-mon[29229]: pgmap v1221: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:29.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:29.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:29.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:29 smithi117 ceph-mon[32835]: pgmap v1221: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:29.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:29.760 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:29 smithi107 ceph-mon[32381]: pgmap v1221: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:29.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:30.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:30.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:30.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:30.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:30.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:30.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:30.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:30.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:30.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:30.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:30.812 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:30.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:31.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:31.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:31.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:31.449 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:31.449 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:31.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:31.480 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:31.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:10:31.481 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:31.494 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:10:31.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:10:31.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:31.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:31 smithi117 ceph-mon[32835]: pgmap v1222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:31.619 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:31.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:31.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:31 smithi086 ceph-mon[29229]: pgmap v1222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:31.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:31.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:31 smithi107 ceph-mon[32381]: pgmap v1222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:32.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:32.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:32.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:32.293 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:32.293 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:32.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:32.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:32.562 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:32 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:10:32.562 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:32.562 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:32 smithi107 ceph-mon[32381]: osdmap e412: 12 total, 3 up, 11 in 2024-08-06T18:10:32.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:32.693 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:32 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:10:32.693 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:32.693 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:32 smithi086 ceph-mon[29229]: osdmap e412: 12 total, 3 up, 11 in 2024-08-06T18:10:32.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:32 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:10:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2823618586' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:32 smithi117 ceph-mon[32835]: osdmap e412: 12 total, 3 up, 11 in 2024-08-06T18:10:32.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:32.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:32.953 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:33.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:33.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:33.319 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:33.319 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:33.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:33.355 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:33.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:33.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:33.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:33 smithi086 ceph-mon[29229]: pgmap v1224: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:33.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:33 smithi117 ceph-mon[32835]: pgmap v1224: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:33.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:33.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:33.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:33 smithi107 ceph-mon[32381]: pgmap v1224: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:33.898 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:33.898 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:34.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:34.122 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:34.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:34.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:10:34.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:34.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:34.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:34.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:34.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:34.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:34.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:34.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:34.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:34.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:35.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:35.072 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:35.072 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:35.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:35.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:35.344 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:35.344 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":412,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:31.482119+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:25.941071+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:35.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:35.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:35.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:35.502 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:10:35.503 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:10:35.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:35.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:35 smithi086 ceph-mon[29229]: pgmap v1225: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:35.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1850472832' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:35 smithi117 ceph-mon[32835]: pgmap v1225: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:35.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1850472832' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:35.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:35.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:35 smithi107 ceph-mon[32381]: pgmap v1225: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1850472832' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:36.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:36.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:36.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:36.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:36.479 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:36.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:36.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:36.699 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:36.699 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:36.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:36.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:10:36.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:37.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:37.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:37.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:37.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:37.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:37.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:37.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:37 smithi117 ceph-mon[32835]: pgmap v1226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:37.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:37 smithi117 ceph-mon[32835]: osdmap e413: 12 total, 3 up, 11 in 2024-08-06T18:10:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:37 smithi107 ceph-mon[32381]: pgmap v1226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:37 smithi107 ceph-mon[32381]: osdmap e413: 12 total, 3 up, 11 in 2024-08-06T18:10:37.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:37 smithi086 ceph-mon[29229]: pgmap v1226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:37 smithi086 ceph-mon[29229]: osdmap e413: 12 total, 3 up, 11 in 2024-08-06T18:10:37.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:38.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:38.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:38.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:38.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:38.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:38.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:38.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:38.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:38.958 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:38.958 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:39.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:39.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:39.311 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:39.311 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:39.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:39.429 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:39.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:39.448 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:39.449 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:39.461 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:39.469 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:39.470 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:39.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:39.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:39.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:39.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:39.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:39.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:39.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:39.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:39 smithi086 ceph-mon[29229]: pgmap v1228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:39.587 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/389061769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:39.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:39.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:39.764 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:39 smithi117 ceph-mon[32835]: pgmap v1228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:39.764 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/389061769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:39.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:39 smithi107 ceph-mon[32381]: pgmap v1228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/389061769' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:40.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:40.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:40.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:40.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:40.481 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:10:40.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:40.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.570 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:40.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:40.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.772 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.772 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.772 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:40.799 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:40.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:40.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:40.837 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:40.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:10:40.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:41.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:41.106 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:41.106 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:41.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:41.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:41.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:41.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:41 smithi117 ceph-mon[32835]: pgmap v1229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:41.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:41 smithi117 ceph-mon[32835]: osdmap e414: 12 total, 3 up, 11 in 2024-08-06T18:10:41.624 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:41 smithi117 ceph-mon[32835]: osdmap e415: 12 total, 3 up, 11 in 2024-08-06T18:10:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:41 smithi107 ceph-mon[32381]: pgmap v1229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:41 smithi107 ceph-mon[32381]: osdmap e414: 12 total, 3 up, 11 in 2024-08-06T18:10:41.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:41 smithi107 ceph-mon[32381]: osdmap e415: 12 total, 3 up, 11 in 2024-08-06T18:10:41.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:41.828 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:41 smithi086 ceph-mon[29229]: pgmap v1229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1529794802' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:10:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:41 smithi086 ceph-mon[29229]: osdmap e414: 12 total, 3 up, 11 in 2024-08-06T18:10:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:41 smithi086 ceph-mon[29229]: osdmap e415: 12 total, 3 up, 11 in 2024-08-06T18:10:41.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:42.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:42.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:42.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:42.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:42.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:42.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:42.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:10:42.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:42.953 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:10:42.953 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:10:42.953 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:10:42.953 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:10:42.953 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:10:42.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:42.999 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:42.999 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:42.999 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:43.000 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:43.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:43.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:43.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:43.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:10:43.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:43.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:43.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:43.544 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:43 smithi086 ceph-mon[29229]: pgmap v1232: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:43.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:43 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:10:43.545 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:43 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:10:43.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:43 smithi107 ceph-mon[32381]: pgmap v1232: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:43 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:10:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:43 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:10:43.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:10:43.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:43 smithi117 ceph-mon[32835]: pgmap v1232: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:43 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:10:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:43 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:10:43.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:44.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:44.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:44.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:44.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:44.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:44.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:44.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:44.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:44 smithi086 ceph-mon[29229]: osdmap e416: 12 total, 3 up, 11 in 2024-08-06T18:10:44.618 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:44 smithi117 ceph-mon[32835]: osdmap e416: 12 total, 3 up, 11 in 2024-08-06T18:10:44.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:44.754 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:44 smithi107 ceph-mon[32381]: osdmap e416: 12 total, 3 up, 11 in 2024-08-06T18:10:44.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:44.919 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:44.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:45.023 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:45.023 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:45.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:45.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:45.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:45.439 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:45.439 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":416,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:43.486041+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:35.823032+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:45.535 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:45 smithi086 ceph-mon[29229]: pgmap v1234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:45.535 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3444360593' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:45.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:45.536 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:45.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:45.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:45.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:45.835 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:45 smithi117 ceph-mon[32835]: pgmap v1234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:45.835 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3444360593' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:45.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:45 smithi107 ceph-mon[32381]: pgmap v1234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3444360593' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:46.066 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:46.066 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:46.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:46.190 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:46.190 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:46.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:46.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:46.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:46.506 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:46.506 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:46.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:46.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:46.771 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:46.771 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:46.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:46 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:10:46.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:46 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:10:46.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:46 smithi086 ceph-mon[29229]: osdmap e417: 12 total, 3 up, 11 in 2024-08-06T18:10:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:46 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:10:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:46 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:10:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:46 smithi117 ceph-mon[32835]: osdmap e417: 12 total, 3 up, 11 in 2024-08-06T18:10:46.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:46.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:46.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:46 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:10:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:46 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:10:46.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:46 smithi107 ceph-mon[32381]: osdmap e417: 12 total, 3 up, 11 in 2024-08-06T18:10:47.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:47.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:47.088 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:47.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:47.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:47.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:47.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:47.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:47.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:47 smithi107 ceph-mon[32381]: pgmap v1235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:47.809 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:47 smithi086 ceph-mon[29229]: pgmap v1235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:47.809 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:47.810 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:47.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:47.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:47 smithi117 ceph-mon[32835]: pgmap v1235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:47.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:48.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:48.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:48.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:48.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:48.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:48.494 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:48.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:48.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:48.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:48.951 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:48.951 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:48.951 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:48.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:48.968 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:48.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:48.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:48.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:48.985 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:48.985 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:48.986 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:48.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:49.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:49.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:49.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:49.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:49.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:49.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:49.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:49.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:49.515 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:10:49.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:49 smithi117 ceph-mon[32835]: pgmap v1237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2108557427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:49.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:49.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:49 smithi086 ceph-mon[29229]: pgmap v1237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:49.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2108557427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:49.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:49.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:49.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:49.844 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:49.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:49.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:49.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:49 smithi107 ceph-mon[32381]: pgmap v1237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2108557427' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:10:50.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:50.110 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:50.111 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:50.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:50.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:50.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:50.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:50.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:50.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:50.649 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:50.649 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:50.650 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:50.650 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:50 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:50.650 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:50.650 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:50 smithi086 ceph-mon[29229]: osdmap e418: 12 total, 3 up, 11 in 2024-08-06T18:10:50.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:50.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:10:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:50 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:50 smithi117 ceph-mon[32835]: osdmap e418: 12 total, 3 up, 11 in 2024-08-06T18:10:50.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:50.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:50.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:50 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:10:50.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2965758476' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:10:50.900 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:50 smithi107 ceph-mon[32381]: osdmap e418: 12 total, 3 up, 11 in 2024-08-06T18:10:51.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:51.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:51.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:51.299 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:51.299 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:51.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:51.534 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:51.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:51.618 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:51.618 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:51.618 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:51 smithi086 ceph-mon[29229]: pgmap v1239: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:51.619 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:51 smithi086 ceph-mon[29229]: osdmap e419: 12 total, 3 up, 11 in 2024-08-06T18:10:51.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:51 smithi107 ceph-mon[32381]: pgmap v1239: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:51.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:51 smithi107 ceph-mon[32381]: osdmap e419: 12 total, 3 up, 11 in 2024-08-06T18:10:51.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:51.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:10:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:51 smithi117 ceph-mon[32835]: pgmap v1239: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:51 smithi117 ceph-mon[32835]: osdmap e419: 12 total, 3 up, 11 in 2024-08-06T18:10:51.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:51.962 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:51.962 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:52.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:52.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:52.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:52.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:52.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:52.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:52.760 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:10:52.760 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:10:52.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:52.902 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:52.902 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:52.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:10:53.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:53.013 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:53.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:53.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:53.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:53.522 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:53.522 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:53.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:53.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:10:53.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:53.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:53 smithi086 ceph-mon[29229]: pgmap v1241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:53.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:10:53.836 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:53 smithi107 ceph-mon[32381]: pgmap v1241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:53.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:53 smithi117 ceph-mon[32835]: pgmap v1241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:53.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:54.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:54.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:54.338 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:54.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:54.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:54.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:54.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:54.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:54.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:54.900 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:54.901 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:54.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:55.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:55.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:55.215 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:55.215 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:55.244 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:55.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:55.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:55.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:55.564 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:55 smithi117 ceph-mon[32835]: pgmap v1242: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:55.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:55.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:55 smithi086 ceph-mon[29229]: pgmap v1242: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:55.787 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:10:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:10:55.787 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:10:55.804 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:10:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:10:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:10:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:10:55.805 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:10:55.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:55.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:10:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:10:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:10:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:10:55.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:10:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:10:55.825 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:10:55.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:55 smithi107 ceph-mon[32381]: pgmap v1242: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:56.036 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:56.036 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:56.037 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:56.037 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:56.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:56.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:56.316 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:56.316 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:56.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:56.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:56.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:56.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:10:56.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:10:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1563183555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:56.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1563183555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1563183555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:10:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:56.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:57.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:57.214 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:57.214 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:57.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:10:57.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:57.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:57.534 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:57.535 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:57.557 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:57.557 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:57 smithi107 ceph-mon[32381]: pgmap v1243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:57.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:57.761 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:10:57.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:10:57.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:57.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:57.827 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:57 smithi117 ceph-mon[32835]: pgmap v1243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:10:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:57 smithi086 ceph-mon[29229]: pgmap v1243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:57.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:57.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:57.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:58.006 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:10:58.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:10:58.171 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:10:58.172 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:10:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:10:58.172 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:10:58.172 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":419,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:10:51.476901+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:10:58.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:10:58.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:58.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:58.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:58.623 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:10:58.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:10:58.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1846418130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:58.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:10:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1846418130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1846418130' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:10:59.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:10:59.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:10:59.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:10:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:10:59.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:10:59.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:59.434 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:10:59.434 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:10:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:10:59.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:10:59.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:10:59.702 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:10:59 smithi107 ceph-mon[32381]: pgmap v1244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:59.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:10:59.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:10:59.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:10:59 smithi086 ceph-mon[29229]: pgmap v1244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:10:59 smithi117 ceph-mon[32835]: pgmap v1244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:10:59.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:00.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:00.091 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:00.091 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:10:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:00.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:00.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:00.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:00.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:00.455 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:00.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:00.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:00.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:00.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:00.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:01.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:01.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:01.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:01.358 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:01.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:01.542 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:01.542 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:01.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:01.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:01.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:01.836 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:01 smithi117 ceph-mon[32835]: pgmap v1245: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:01.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:01 smithi086 ceph-mon[29229]: pgmap v1245: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:01.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:01.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:01.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:01 smithi107 ceph-mon[32381]: pgmap v1245: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:02.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:02.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:02.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:02.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:02.546 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:02.546 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:02.550 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:11:02.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:02.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:02.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:02.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:02.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:02.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:02.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:03.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:03.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:03.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:03.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:03.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:03.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:03.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:03.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:03.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:03.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:03.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:03 smithi086 ceph-mon[29229]: pgmap v1246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:03.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:03 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:03.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:03.566 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:03 smithi086 ceph-mon[29229]: osdmap e420: 12 total, 3 up, 11 in 2024-08-06T18:11:03.665 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:03.666 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:03.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:03.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:03.681 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:03.682 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:03.682 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:03.682 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:03.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:03.699 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:03.700 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:03.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:03 smithi117 ceph-mon[32835]: pgmap v1246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:03.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:03 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:03.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:03.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:03 smithi117 ceph-mon[32835]: osdmap e420: 12 total, 3 up, 11 in 2024-08-06T18:11:03.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:03.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:03 smithi107 ceph-mon[32381]: pgmap v1246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:03.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:03 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:03.846 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3203972222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:03.846 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:03 smithi107 ceph-mon[32381]: osdmap e420: 12 total, 3 up, 11 in 2024-08-06T18:11:03.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:04.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:04.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:04.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:04.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:04.426 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:04.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:04.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:04.494 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:04.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:04.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/970592903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:04.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/970592903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:04.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/970592903' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:04.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:04.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:05.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:05.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:05.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:05.319 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:05.320 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:05.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:05.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:05.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:05 smithi086 ceph-mon[29229]: pgmap v1248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:05.581 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:05 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:11:05.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:05.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:05.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:05.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:05.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:05.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:05 smithi117 ceph-mon[32835]: pgmap v1248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:05.842 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:05 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:11:05.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:05.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:05.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:05.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:05.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:05 smithi107 ceph-mon[32381]: pgmap v1248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:05 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:11:06.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:06.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:06.239 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:06.239 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:06.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:06.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:11:06.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:06.553 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:06.553 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:06.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:06.651 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:06.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:06.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:11:06.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:11:06.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:11:06.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:06 smithi086 ceph-mon[29229]: osdmap e421: 12 total, 3 up, 11 in 2024-08-06T18:11:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:11:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:11:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:11:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:06 smithi117 ceph-mon[32835]: osdmap e421: 12 total, 3 up, 11 in 2024-08-06T18:11:06.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:11:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:11:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:11:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:06 smithi107 ceph-mon[32381]: osdmap e421: 12 total, 3 up, 11 in 2024-08-06T18:11:07.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:07.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:07.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:07.292 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:07.293 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:07.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:11:07.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:07.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:07.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:07.682 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:07 smithi086 ceph-mon[29229]: pgmap v1249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:07.712 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:07.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:11:07.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:07 smithi117 ceph-mon[32835]: pgmap v1249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:07.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:07.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:07 smithi107 ceph-mon[32381]: pgmap v1249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:08.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:08.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:08.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:08.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:08.481 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:08.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:08.700 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:08.704 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:08.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:08.870 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:08.871 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":421,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:06.480856+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:09:51.359034+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:08.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:09.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:09.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:09.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:09.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:09.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:09.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:09.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:09 smithi107 ceph-mon[32381]: pgmap v1251: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:09.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1941037888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:09.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:09 smithi086 ceph-mon[29229]: pgmap v1251: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:09.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1941037888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:09.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:09 smithi117 ceph-mon[32835]: pgmap v1251: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1941037888' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:09.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:09.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:09.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:10.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:10.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:10.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:10.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:10.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:10.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:10.808 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:10.809 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:10.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:11.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:11.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:11.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:11.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:11.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:11.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:11 smithi117 ceph-mon[32835]: pgmap v1252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:11.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:11.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:11 smithi086 ceph-mon[29229]: pgmap v1252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:11.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:11.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:11 smithi107 ceph-mon[32381]: pgmap v1252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:12.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:12.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:12.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:12.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:12.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:12.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:12.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:12.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:12.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:12.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:12.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:12.822 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:12.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:13.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:13.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:13.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:13.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:13.246 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:13.246 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:13.308 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:13.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:13.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:13.562 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:13.563 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:13.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:13.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:13 smithi086 ceph-mon[29229]: pgmap v1253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:13 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:11:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:13 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:11:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:13 smithi117 ceph-mon[32835]: pgmap v1253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:13 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:11:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:13 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:11:13.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:13.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:11:13.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:13.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:13 smithi107 ceph-mon[32381]: pgmap v1253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:13 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:11:13.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:13 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:11:14.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:14.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:14.173 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:14.173 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:14.173 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:14.173 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:14.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:14.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:14.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:14.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:14.846 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:14.847 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:14.847 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:14.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:14.857 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:14.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:14.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:14.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:14.859 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:14.859 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:14.859 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:14.864 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:14.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:14.992 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:14.992 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:15.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:15.134 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:15.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:15.255 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:15.256 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:15.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:15.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:11:15.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:15.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:15.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:15.673 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:15.673 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:15.673 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:15 smithi107 ceph-mon[32381]: pgmap v1254: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:15.673 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3083176513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:15.674 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.674 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:15.674 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:15.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:15 smithi086 ceph-mon[29229]: pgmap v1254: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:15.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3083176513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:15 smithi117 ceph-mon[32835]: pgmap v1254: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3083176513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:15.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:15.939 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:15.939 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:16.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:16.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:16.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:16.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:16.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:16.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:16.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:16.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:16 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:16.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:16.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:16 smithi086 ceph-mon[29229]: osdmap e422: 12 total, 3 up, 11 in 2024-08-06T18:11:16.742 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:16 smithi086 ceph-mon[29229]: osdmap e423: 12 total, 3 up, 11 in 2024-08-06T18:11:16.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:16 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:16 smithi117 ceph-mon[32835]: osdmap e422: 12 total, 3 up, 11 in 2024-08-06T18:11:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:16 smithi117 ceph-mon[32835]: osdmap e423: 12 total, 3 up, 11 in 2024-08-06T18:11:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:16 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/26497141' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:16 smithi107 ceph-mon[32381]: osdmap e422: 12 total, 3 up, 11 in 2024-08-06T18:11:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:16 smithi107 ceph-mon[32381]: osdmap e423: 12 total, 3 up, 11 in 2024-08-06T18:11:16.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:17.008 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:17.008 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:17.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:17.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:17.364 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:11:17.365 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:11:17.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:17.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:17.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:17.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:17.653 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:17.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:17.810 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:17 smithi117 ceph-mon[32835]: pgmap v1256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:17.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:17 smithi086 ceph-mon[29229]: pgmap v1256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:17.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:17.946 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:17 smithi107 ceph-mon[32381]: pgmap v1256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:18.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:18.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:11:18.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:18.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:18.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:18.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:11:18.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:18.619 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:18.621 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:18.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:18.811 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:18.811 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:18.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:18.890 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:18.890 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:19.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:19.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:19.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:19.289 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:19.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:19.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:19.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:19.685 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:19.685 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:19.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:19 smithi086 ceph-mon[29229]: pgmap v1258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:19.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:19 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:11:19.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:19 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:11:19.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:19.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:19 smithi107 ceph-mon[32381]: pgmap v1258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:19.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:19 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:11:19.838 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:19 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:11:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:19 smithi117 ceph-mon[32835]: pgmap v1258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:19 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:11:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:19 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:11:19.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:20.010 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:20.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:20.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:20.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:20.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:20.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:20.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:20.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:20.650 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:20 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:11:20.650 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:20 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:11:20.650 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:20 smithi117 ceph-mon[32835]: osdmap e424: 12 total, 3 up, 11 in 2024-08-06T18:11:20.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:20.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:20 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:11:20.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:20 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:11:20.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:20 smithi086 ceph-mon[29229]: osdmap e424: 12 total, 3 up, 11 in 2024-08-06T18:11:20.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:20 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:11:20.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:20 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:11:20.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:20 smithi107 ceph-mon[32381]: osdmap e424: 12 total, 3 up, 11 in 2024-08-06T18:11:20.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:11:20.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:21.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:21.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:21.186 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:21.186 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:21.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:21.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:21.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:21.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:21.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:21.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:21.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:21.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:21 smithi117 ceph-mon[32835]: pgmap v1260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:21.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:21 smithi117 ceph-mon[32835]: osdmap e425: 12 total, 3 up, 11 in 2024-08-06T18:11:21.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:21 smithi086 ceph-mon[29229]: pgmap v1260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:21 smithi086 ceph-mon[29229]: osdmap e425: 12 total, 3 up, 11 in 2024-08-06T18:11:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:21 smithi107 ceph-mon[32381]: pgmap v1260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:21 smithi107 ceph-mon[32381]: osdmap e425: 12 total, 3 up, 11 in 2024-08-06T18:11:21.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:22.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:22.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:22.259 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:22.259 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:22.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:22.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:22.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:22.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:22.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:22.684 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:22.688 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:11:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:11:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:11:22.688 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:11:22.688 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:11:22.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:22.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:22.935 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:22.935 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:23.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:23.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:23.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:23.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:23.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:23.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:23 smithi086 ceph-mon[29229]: pgmap v1262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:23.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:23 smithi086 ceph-mon[29229]: osdmap e426: 12 total, 3 up, 11 in 2024-08-06T18:11:23.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:23.746 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:23 smithi107 ceph-mon[32381]: pgmap v1262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:23.746 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:23 smithi107 ceph-mon[32381]: osdmap e426: 12 total, 3 up, 11 in 2024-08-06T18:11:23.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:23 smithi117 ceph-mon[32835]: pgmap v1262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:23 smithi117 ceph-mon[32835]: osdmap e426: 12 total, 3 up, 11 in 2024-08-06T18:11:23.953 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:23.953 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:23.953 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:23.954 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":426,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:22.487164+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:23.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:24.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:24.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:24.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:11:24.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:24.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:24.550 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/812525630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:24.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:24.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/812525630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:24.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:24.948 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/812525630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:25.027 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:25.027 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:25.027 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:25.028 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:25.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:25.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:25.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:25.404 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:25.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:25.494 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:11:25.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:25.621 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:25 smithi107 ceph-mon[32381]: pgmap v1264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:25.622 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:11:25.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:25.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:25 smithi117 ceph-mon[32835]: pgmap v1264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:25.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:11:25.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:25 smithi086 ceph-mon[29229]: pgmap v1264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:25.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:11:25.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:25.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:25.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:25.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:26.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:26.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:26.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:26.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:26.553 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:26.553 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:26.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:26.567 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:26.568 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:26.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:26.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:26.568 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:26.586 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:26.586 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:26.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:26.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:26.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:26.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:26.587 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:26 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:11:26.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:26 smithi117 ceph-mon[32835]: osdmap e427: 12 total, 3 up, 11 in 2024-08-06T18:11:26.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:26.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:26.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:26 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:26.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:11:26.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:26 smithi086 ceph-mon[29229]: osdmap e427: 12 total, 3 up, 11 in 2024-08-06T18:11:26.817 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:26.817 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:26.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:26 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3937521949' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:11:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:26 smithi107 ceph-mon[32381]: osdmap e427: 12 total, 3 up, 11 in 2024-08-06T18:11:27.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:27.076 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:27.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:27.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:27.077 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:27.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:27.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:27.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:27.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:27.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:27 smithi086 ceph-mon[29229]: pgmap v1266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:27.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/404370638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:27.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.607 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.607 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:27.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:27 smithi107 ceph-mon[32381]: pgmap v1266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/404370638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:27.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:27 smithi117 ceph-mon[32835]: pgmap v1266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/404370638' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:27.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:28.012 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:28.012 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:28.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:28.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:28.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:28.282 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:28.282 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:28.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:28.520 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:28.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:28.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:28.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:28.829 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:28.830 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:28.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:28.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:28.955 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:28.955 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:29.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:11:29.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:29.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:29.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:29.363 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:11:29.363 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:11:29.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:29.515 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:29.515 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:29.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:29.617 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:29.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:29.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:29.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:29 smithi086 ceph-mon[29229]: pgmap v1267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:29 smithi117 ceph-mon[32835]: pgmap v1267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:29 smithi107 ceph-mon[32381]: pgmap v1267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:29.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:30.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:30.143 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:30.144 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:30.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:11:30.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:30.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:30.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:30.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:30.447 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:30.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:30.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:30.783 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:30.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:30.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:30.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:30.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:11:30.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:31.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:31.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:31.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:31.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:31.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:31.327 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:31.328 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:31.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:31.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:31.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:31 smithi086 ceph-mon[29229]: pgmap v1268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:31.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:31.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:31 smithi107 ceph-mon[32381]: pgmap v1268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:31.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:31.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:31.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:31 smithi117 ceph-mon[32835]: pgmap v1268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:31.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:31.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:31.980 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:31.992 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:32.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:32.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:32.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:32.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:32.594 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:32.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:32.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:32.762 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:32.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:11:32.775 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:32.775 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":427,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:25.495137+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:32.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:33.008 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:33.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:33.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:33.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:33.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:33.445 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:33.445 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:33.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:33.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:33.700 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:33 smithi107 ceph-mon[32381]: pgmap v1269: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:33.700 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1458380769' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:33.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:33.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:33 smithi086 ceph-mon[29229]: pgmap v1269: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:33.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1458380769' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:33 smithi117 ceph-mon[32835]: pgmap v1269: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1458380769' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:33.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:34.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:34.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:34.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:34.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:34.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:34.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:34.629 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:34.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:34.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:34.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:34.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:34.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:35.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:35.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:35.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:35.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:35.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:35.310 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:35.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:11:35.439 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:35.440 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:35.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:35.558 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:35.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:35.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:35 smithi117 ceph-mon[32835]: pgmap v1270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:35.680 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:35 smithi107 ceph-mon[32381]: pgmap v1270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:35.692 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:35 smithi086 ceph-mon[29229]: pgmap v1270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:35.693 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:35.693 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:35.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:35.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:35.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:35.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:35.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:35.808 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:35.808 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:35.808 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:35.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:35.825 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:35.826 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:35.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:35.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:35.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:35.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:35.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:35.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:35.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:36.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:36.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:36.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:36.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:36.547 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:36.547 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:36.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:36.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:36.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:36.790 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4025116707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:36.790 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.790 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4025116707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4025116707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:36.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:36.938 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:37.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:37.135 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:37.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:37.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:37.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:37.455 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:37.455 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:37.531 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:11:37.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:37.666 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:37.666 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:37.666 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:37.667 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:37 smithi086 ceph-mon[29229]: pgmap v1271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:37.667 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:37.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:37 smithi107 ceph-mon[32381]: pgmap v1271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:37.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:11:37.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:37 smithi117 ceph-mon[32835]: pgmap v1271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:37.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:38.029 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:38.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:38.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:38.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:38.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:38.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:11:38.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:38.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:38 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:38.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:38.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:38 smithi107 ceph-mon[32381]: osdmap e428: 12 total, 3 up, 11 in 2024-08-06T18:11:38.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:38 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:38.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:38.741 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:38 smithi086 ceph-mon[29229]: osdmap e428: 12 total, 3 up, 11 in 2024-08-06T18:11:38.741 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:38.741 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:38.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:38.811 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:38 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1900232076' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:38 smithi117 ceph-mon[32835]: osdmap e428: 12 total, 3 up, 11 in 2024-08-06T18:11:38.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:39.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:39.017 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:39.018 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:39.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:39.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:39.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:39.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:39.649 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:39.675 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:39.675 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":428,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:37.532161+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:39.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:39.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:39 smithi107 ceph-mon[32381]: pgmap v1273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:39.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:39 smithi086 ceph-mon[29229]: pgmap v1273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:39.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:39.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:39.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:39 smithi117 ceph-mon[32835]: pgmap v1273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:39.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:40.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:40.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:40.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:40.378 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:40.378 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:40.405 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:40.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:11:40.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:40.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:40.655 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:40.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/166956412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:40.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:40.772 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/166956412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:40.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:11:40.801 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/166956412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:40.801 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:40.801 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:40.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:41.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:41.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:41.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:41.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:41.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:41.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:41.445 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:41.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:41.572 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:41 smithi117 ceph-mon[32835]: pgmap v1274: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:41.572 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:41 smithi117 ceph-mon[32835]: osdmap e429: 12 total, 3 up, 11 in 2024-08-06T18:11:41.650 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:41.650 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:41.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:41.667 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:41.668 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:41.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:41.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:41.668 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:41.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:41.685 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:41 smithi086 ceph-mon[29229]: pgmap v1274: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:41.686 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:41 smithi086 ceph-mon[29229]: osdmap e429: 12 total, 3 up, 11 in 2024-08-06T18:11:41.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:41.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:41.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:41 smithi107 ceph-mon[32381]: pgmap v1274: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:41.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:41 smithi107 ceph-mon[32381]: osdmap e429: 12 total, 3 up, 11 in 2024-08-06T18:11:41.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:41.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:42.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:42.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:42.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:42.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:42.230 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:42.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:42.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:42.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:42.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:42.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/427183771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:42.749 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:42.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:42.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:42.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:42.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:42.766 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/427183771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:42.766 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.766 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.766 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/427183771' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:42.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.775 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:42.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:42.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:42.951 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:42.952 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:11:43.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:43.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:43.197 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:43.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:43.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:43.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:43.673 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:43 smithi107 ceph-mon[32381]: pgmap v1276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:43.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:43.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:43 smithi086 ceph-mon[29229]: pgmap v1276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:43.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:43.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:43 smithi117 ceph-mon[32835]: pgmap v1276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:43.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:44.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:44.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:44.217 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:44.330 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:44.331 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:44.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:44.446 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:44.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:44.562 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:11:44.604 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:44.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:44.703 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:44.703 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:44.703 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:44.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:44.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:11:44.813 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:11:44.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:45.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:45.019 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:45.019 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:45.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:45.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:45.404 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:45.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:11:45.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:45.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:45.563 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:45.563 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:45.563 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:45.563 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:45.652 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:45.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:45 smithi107 ceph-mon[32381]: pgmap v1277: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:45 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:45.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:45.714 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:45 smithi107 ceph-mon[32381]: osdmap e430: 12 total, 3 up, 11 in 2024-08-06T18:11:45.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:45.839 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:45 smithi117 ceph-mon[32835]: pgmap v1277: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:45 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:45 smithi117 ceph-mon[32835]: osdmap e430: 12 total, 3 up, 11 in 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:45 smithi086 ceph-mon[29229]: pgmap v1277: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:45 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:11:45.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2243074156' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:11:45.841 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:45 smithi086 ceph-mon[29229]: osdmap e430: 12 total, 3 up, 11 in 2024-08-06T18:11:45.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:46.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:46.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:46.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:46.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:46.491 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:46.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:46.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:46.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:47.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:47.116 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:47.116 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:47.116 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:47.116 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:47.122 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:47.122 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":431,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:46.500524+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:42.958414+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:47.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:47.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:47.354 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:11:47.355 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:11:47.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:47.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:47.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:47.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:47.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:47.634 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:47 smithi086 ceph-mon[29229]: pgmap v1279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:47.634 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:47 smithi086 ceph-mon[29229]: osdmap e431: 12 total, 3 up, 11 in 2024-08-06T18:11:47.634 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:47 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:11:47.634 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:47 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:11:47.635 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3250532558' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:47.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:47.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:47 smithi107 ceph-mon[32381]: pgmap v1279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:47.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:47 smithi107 ceph-mon[32381]: osdmap e431: 12 total, 3 up, 11 in 2024-08-06T18:11:47.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:47 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:11:47.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:47 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:11:47.696 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3250532558' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:47.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:47.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:47 smithi117 ceph-mon[32835]: pgmap v1279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:47.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:47 smithi117 ceph-mon[32835]: osdmap e431: 12 total, 3 up, 11 in 2024-08-06T18:11:47.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:47 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:11:47.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:47 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:11:47.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3250532558' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:47.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:11:47.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:48.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:48.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:11:48.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:48.326 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:48.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:48.359 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:48.359 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:48.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:48.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:48.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:48 smithi086 ceph-mon[29229]: osdmap e432: 12 total, 3 up, 11 in 2024-08-06T18:11:48.624 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:48 smithi107 ceph-mon[32381]: osdmap e432: 12 total, 3 up, 11 in 2024-08-06T18:11:48.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:48.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:48 smithi117 ceph-mon[32835]: osdmap e432: 12 total, 3 up, 11 in 2024-08-06T18:11:48.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:48.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:48.977 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:48.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:49.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:49.213 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:49.214 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:49.270 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:49.270 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:49.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:49.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:49.529 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:49.529 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:49.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:49.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:49.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:49.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:49 smithi107 ceph-mon[32381]: pgmap v1282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:49.826 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:49 smithi086 ceph-mon[29229]: pgmap v1282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:49.826 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:49.826 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:49.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:49.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:49.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:49 smithi117 ceph-mon[32835]: pgmap v1282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:49.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:50.035 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:11:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:11:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:11:50.035 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:11:50.035 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:11:50.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:50.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:50.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:50.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:50.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:50.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:50.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:50.627 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/766106255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:50.628 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.628 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.628 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:50.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:50.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:50.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/766106255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:50.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:11:50.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:50.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:50.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/766106255' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:50.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:51.043 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:51.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:51.179 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:51.180 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:51.208 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:51.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:51.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:51.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:51.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:51.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:51.685 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:51 smithi117 ceph-mon[32835]: pgmap v1283: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:51.685 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:51 smithi117 ceph-mon[32835]: osdmap e433: 12 total, 3 up, 11 in 2024-08-06T18:11:51.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:51.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:51.816 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:51 smithi107 ceph-mon[32381]: pgmap v1283: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:51.816 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:51 smithi107 ceph-mon[32381]: osdmap e433: 12 total, 3 up, 11 in 2024-08-06T18:11:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:51 smithi086 ceph-mon[29229]: pgmap v1283: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:51 smithi086 ceph-mon[29229]: osdmap e433: 12 total, 3 up, 11 in 2024-08-06T18:11:51.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:52.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:52.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:52.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:52.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:52.530 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:52.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:52.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:52.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:52.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:52.584 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:293,from:0,reservation_nonce:0)", 2024-08-06T18:11:52.690 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 592.46750306199999, 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000202034, 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.691 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222754+0000", 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222755+0000", 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0979999999999999e-06 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.692 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222758+0000", 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5659999999999998e-06 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222770+0000", 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1504e-05 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222779+0000", 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.4970000000000007e-06 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.693 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222841+0000", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.2020999999999995e-05 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222854+0000", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2199e-05 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.222956+0000", 2024-08-06T18:11:52.694 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000102149 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 592.46673439599999, 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1451000000000003e-05, 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.695 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.696 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223522+0000", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223523+0000", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8799999999999998e-07 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223526+0000", 2024-08-06T18:11:52.697 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7580000000000001e-06 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223532+0000", 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5570000000000001e-06 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223537+0000", 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7469999999999996e-06 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.698 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223581+0000", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3791000000000001e-05 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223587+0000", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.9290000000000003e-06 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:02:00.223604+0000", 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7280999999999999e-05 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.699 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:311,from:0,reservation_nonce:0)", 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 521.46055459700005, 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026581800000000002, 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.700 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229702+0000", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229704+0000", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.66e-06 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.701 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229709+0000", 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.8219999999999997e-06 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229722+0000", 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3616e-05 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229748+0000", 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5868999999999999e-05 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.702 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229802+0000", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4267000000000001e-05 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229823+0000", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0241000000000002e-05 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.229968+0000", 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014534300000000001 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.703 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 521.45964181199997, 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000105858, 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.704 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230615+0000", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230616+0000", 2024-08-06T18:11:52.705 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0299999999999998e-07 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230620+0000", 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8360000000000002e-06 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230629+0000", 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7070000000000004e-06 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.706 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230635+0000", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4620000000000004e-06 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230679+0000", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3914e-05 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230698+0000", 2024-08-06T18:11:52.707 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9364e-05 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:03:11.230721+0000", 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2872000000000001e-05 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:328,from:0,reservation_nonce:0)", 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:04:25.651973+0000", 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 447.03828367599999, 2024-08-06T18:11:52.708 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000192888, 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651973+0000", 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651973+0000", 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.709 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651974+0000", 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.133e-06 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651979+0000", 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3599999999999998e-06 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.710 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651989+0000", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0257e-05 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.651996+0000", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3180000000000001e-06 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652041+0000", 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.5284000000000002e-05 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.711 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652055+0000", 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3175e-05 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652166+0000", 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011136100000000001 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.712 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:04:25.652785+0000", 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 447.037471369, 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1859000000000003e-05, 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652785+0000", 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.713 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652785+0000", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652786+0000", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.8199999999999999e-07 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652789+0000", 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3560000000000001e-06 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.714 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652799+0000", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.7929999999999992e-06 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652804+0000", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7589999999999997e-06 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652842+0000", 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8248999999999999e-05 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.715 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652849+0000", 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4080000000000003e-06 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:04:25.652867+0000", 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8612000000000001e-05 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.716 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:387,from:0,reservation_nonce:0)", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 186.13723709300001, 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028996399999999998, 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.717 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553021+0000", 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.345e-06 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553027+0000", 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1750000000000002e-06 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.718 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553041+0000", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3604000000000001e-05 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553068+0000", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7019999999999999e-05 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553168+0000", 2024-08-06T18:11:52.719 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100074 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553185+0000", 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7234000000000001e-05 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553310+0000", 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012451199999999999 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.720 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:11:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:11:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 186.13640005900001, 2024-08-06T18:11:52.721 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5891999999999996e-05, 2024-08-06T18:11:52.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.722 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7400000000000003e-07 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.723 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553861+0000", 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3569999999999998e-06 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553874+0000", 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3671e-05 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553880+0000", 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4399999999999996e-06 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.724 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553923+0000", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3322000000000003e-05 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553930+0000", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4479999999999999e-06 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553953+0000", 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3079999999999999e-05 2024-08-06T18:11:52.725 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:423,from:0,reservation_nonce:0)", 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.671210932000001, 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030840499999999999, 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019047+0000", 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.44e-06 2024-08-06T18:11:52.727 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019054+0000", 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9269999999999999e-06 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019070+0000", 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5523e-05 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.728 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019096+0000", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6038999999999999e-05 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019163+0000", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6780999999999996e-05 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019184+0000", 2024-08-06T18:11:52.729 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1480000000000001e-05 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019354+0000", 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017021500000000001 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:11:52.730 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 33.670306848999999, 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115958, 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.731 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019951+0000", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6499999999999999e-07 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019966+0000", 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5464999999999998e-05 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.732 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019980+0000", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3532e-05 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019987+0000", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1060000000000002e-06 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020023+0000", 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6131999999999999e-05 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.733 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020030+0000", 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3780000000000003e-06 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020066+0000", 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5679999999999997e-05 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:11:52.734 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:11:52.735 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:11:52.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:52.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:11:52.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:53.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:53.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:11:53.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:53.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:53.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:53.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:53.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:11:53.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:53.813 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:53 smithi117 ceph-mon[32835]: pgmap v1285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:53.813 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:53 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:53.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:53.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:53 smithi117 ceph-mon[32835]: osdmap e434: 12 total, 3 up, 11 in 2024-08-06T18:11:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:53 smithi086 ceph-mon[29229]: pgmap v1285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:53 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:53 smithi086 ceph-mon[29229]: osdmap e434: 12 total, 3 up, 11 in 2024-08-06T18:11:53.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:53.925 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:53.945 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:53.945 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:53.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:53 smithi107 ceph-mon[32381]: pgmap v1285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:53.945 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:53 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:11:53.946 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2667043901' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:11:53.946 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:53 smithi107 ceph-mon[32381]: osdmap e434: 12 total, 3 up, 11 in 2024-08-06T18:11:54.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:54.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:54.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:54.323 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:54.343 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:11:54.344 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":434,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:11:52.580595+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:10:46.005988+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:11:54.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:11:54.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:54.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:54.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2878979951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:54.794 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:11:54.794 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:11:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:11:54.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:11:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2878979951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2878979951' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:11:54.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:55.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:55.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:55.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:55.407 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:55.407 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:55.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:55.587 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:55 smithi107 ceph-mon[32381]: pgmap v1287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:55.600 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:55.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:55.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:11:55.684 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:55 smithi086 ceph-mon[29229]: pgmap v1287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:55.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:55 smithi117 ceph-mon[32835]: pgmap v1287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:55.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:56.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:56.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:11:56.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:56.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:56.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:56.327 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:56.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:11:56.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:56.460 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:11:56.460 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:11:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:11:56.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:56.577 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:56.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:56.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:56.878 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:11:56.879 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:11:56.879 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:11:56.884 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:56.900 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:11:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:11:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:11:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:11:56.902 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:11:56.920 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:11:56.920 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:11:56.920 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:56.921 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:56.921 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:11:56.921 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:11:56.921 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:11:56.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:11:56.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:11:56.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:11:57.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:57.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:57.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:57.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:57.504 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:57.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:57.539 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:57 smithi107 ceph-mon[32381]: pgmap v1288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:57.539 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:57 smithi107 ceph-mon[32381]: osdmap e435: 12 total, 3 up, 11 in 2024-08-06T18:11:57.539 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3754602592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:57.539 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.539 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.647 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:57.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:57.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:57 smithi117 ceph-mon[32835]: pgmap v1288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:57.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:57 smithi117 ceph-mon[32835]: osdmap e435: 12 total, 3 up, 11 in 2024-08-06T18:11:57.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3754602592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:57.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:57 smithi086 ceph-mon[29229]: pgmap v1288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:57 smithi086 ceph-mon[29229]: osdmap e435: 12 total, 3 up, 11 in 2024-08-06T18:11:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3754602592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:11:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:11:57.917 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:11:57.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:11:58.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:58.169 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:11:58.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:58.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:11:58.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:58.601 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:58.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:58.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:58.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:58.758 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:58.758 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:58.839 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:11:58.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:11:58.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:11:58.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:59.084 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:11:59.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:11:59.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:11:59.321 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:11:59.321 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:11:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:11:59.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:11:59.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:11:59.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:11:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:59 smithi086 ceph-mon[29229]: pgmap v1290: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:59 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:11:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:59 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:11:59.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:11:59 smithi086 ceph-mon[29229]: osdmap e436: 12 total, 3 up, 11 in 2024-08-06T18:11:59.628 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:11:59.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:11:59.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:59 smithi107 ceph-mon[32381]: pgmap v1290: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:59.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:59 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:11:59.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:59 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:11:59.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:11:59 smithi107 ceph-mon[32381]: osdmap e436: 12 total, 3 up, 11 in 2024-08-06T18:11:59.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:11:59.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:59 smithi117 ceph-mon[32835]: pgmap v1290: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:11:59.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:59 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:11:59.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:59 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:11:59.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:11:59 smithi117 ceph-mon[32835]: osdmap e436: 12 total, 3 up, 11 in 2024-08-06T18:11:59.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:11:59.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:00.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:00.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:00.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:00.349 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:00.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:00.520 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:12:00.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:00.614 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:00.699 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:00.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:00.765 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:00.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:00.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:01.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:01.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:01.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:01.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:01.411 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:01.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:12:01.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:01 smithi086 ceph-mon[29229]: pgmap v1292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:01 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:01 smithi086 ceph-mon[29229]: osdmap e437: 12 total, 3 up, 11 in 2024-08-06T18:12:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:01 smithi117 ceph-mon[32835]: pgmap v1292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:01 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:01.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:01 smithi117 ceph-mon[32835]: osdmap e437: 12 total, 3 up, 11 in 2024-08-06T18:12:01.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:01.669 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:01.685 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:01 smithi107 ceph-mon[32381]: pgmap v1292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:01.685 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:01 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:01.685 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1087641263' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:01.685 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:01 smithi107 ceph-mon[32381]: osdmap e437: 12 total, 3 up, 11 in 2024-08-06T18:12:01.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:01.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:02.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:02.073 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:02.073 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:02.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:02.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:02.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:02.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:02.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:02.598 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:02.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:02.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:02.915 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":437,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:00.513163+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:02.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:03.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:03.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:03.150 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:03.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:03.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:03.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:03.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:03.657 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:03 smithi107 ceph-mon[32381]: pgmap v1294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:03.657 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3942312324' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:03.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:03.786 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:03 smithi117 ceph-mon[32835]: pgmap v1294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:03.786 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3942312324' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:03.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:03 smithi086 ceph-mon[29229]: pgmap v1294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:03.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3942312324' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:03.834 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:03.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:12:03.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:03.928 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:03.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:03.945 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:03.946 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:03.963 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:03.963 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:03.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:03.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:03.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:03.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:03.964 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:04.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:04.082 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:04.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:04.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:04.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:04.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:12:04.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:04.579 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/860735276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:04.579 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.579 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:04.705 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:04.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/860735276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:04.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:04.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/860735276' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:04.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:04.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:04.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:04.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:05.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:05.146 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:05.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:05.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:05.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:05.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:05.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:05.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:05.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:05.691 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:05.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:05.766 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:05.766 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:05 smithi086 ceph-mon[29229]: pgmap v1295: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:05.766 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:05.766 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:05.785 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:05.786 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:05 smithi117 ceph-mon[32835]: pgmap v1295: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:05.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:05.901 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:05.901 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:05 smithi107 ceph-mon[32381]: pgmap v1295: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:06.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:06.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:06.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:06.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:06.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:06.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:06.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:12:06.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:06.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:06.594 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:12:06.594 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:12:06.594 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:12:06.594 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:06 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:12:06.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:06.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:12:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:12:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:12:06.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:06 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:12:06.638 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:06.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:06.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:06.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:06.947 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:06.947 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:06.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:12:06.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:12:06.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:12:06.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:06 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:12:07.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:07.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:12:07.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:07.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:07.397 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:07.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:07.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:07.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:07.542 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:12:07.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:07.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:07.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:07 smithi107 ceph-mon[32381]: pgmap v1296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:07.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:07.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:07.765 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:07 smithi117 ceph-mon[32835]: pgmap v1296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:07.765 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:07.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:07 smithi086 ceph-mon[29229]: pgmap v1296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:07.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:07.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:07.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:08.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:08.130 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:08.130 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:08.210 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:08.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:08.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:08.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:08.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:08.723 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:08.723 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:08.723 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:08 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:08.723 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:08.723 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:08 smithi086 ceph-mon[29229]: osdmap e438: 12 total, 3 up, 11 in 2024-08-06T18:12:08.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:08 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:08 smithi117 ceph-mon[32835]: osdmap e438: 12 total, 3 up, 11 in 2024-08-06T18:12:08.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:08.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:12:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:08 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/694659809' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:08 smithi107 ceph-mon[32381]: osdmap e438: 12 total, 3 up, 11 in 2024-08-06T18:12:08.979 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:09.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:09.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:09.242 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:09.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:09.318 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:09.318 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:09.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:09.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:09.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:09.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:09.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:09 smithi086 ceph-mon[29229]: pgmap v1298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:09.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:09.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:09.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:09 smithi117 ceph-mon[32835]: pgmap v1298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:09.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:09.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:09.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:09 smithi107 ceph-mon[32381]: pgmap v1298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:10.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:10.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:10.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:10.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:12:10.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:10.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:10.678 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:10.678 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:10.678 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:10.696 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:10.697 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:10.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:10.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:10.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:10.715 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:10.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:10.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:10.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:10.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:10.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:10.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:10.722 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:10.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:10.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:11.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:11.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:11.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:11.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:11.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:11.403 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:11.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:12:11.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:11.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:11.652 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:11.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:11.761 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:11 smithi117 ceph-mon[32835]: pgmap v1299: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:11.761 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2134395859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:11.761 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.761 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.761 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:11 smithi117 ceph-mon[32835]: osdmap e439: 12 total, 3 up, 11 in 2024-08-06T18:12:11.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:11.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:11 smithi086 ceph-mon[29229]: pgmap v1299: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:11.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2134395859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:11.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:11 smithi086 ceph-mon[29229]: osdmap e439: 12 total, 3 up, 11 in 2024-08-06T18:12:11.944 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:11.944 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:11 smithi107 ceph-mon[32381]: pgmap v1299: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2134395859' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:11 smithi107 ceph-mon[32381]: osdmap e439: 12 total, 3 up, 11 in 2024-08-06T18:12:11.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:11.984 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:12.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:12.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:12.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:12.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:12.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:12.666 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:12.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:12.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:12.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:12.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:12.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:12.881 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:12.881 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":439,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:11.508197+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:19.387369+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:12.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:13.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:13.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:13.178 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:13.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:13.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:13.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:13.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:13.570 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:13.592 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:13 smithi117 ceph-mon[32835]: pgmap v1301: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:13.592 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3959037471' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:13.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:13.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:13 smithi107 ceph-mon[32381]: pgmap v1301: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:13.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3959037471' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:13.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:13 smithi086 ceph-mon[29229]: pgmap v1301: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:13.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3959037471' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:13.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:13.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:12:13.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:12:13.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:14.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:14.158 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:14.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:14.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:14.391 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:14.484 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:14.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:14.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:14.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:14.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:15.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:15.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:15.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:15.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:15.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:15.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:15.560 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:12:15.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:15.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:15.699 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:15 smithi117 ceph-mon[32835]: pgmap v1302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:15.699 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:15.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:15.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:15 smithi086 ceph-mon[29229]: pgmap v1302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:15.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:15.845 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:15.845 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:15.909 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:15.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:15 smithi107 ceph-mon[32381]: pgmap v1302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:16.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:16.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:16.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:16.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:16.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:16.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:16.234 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:16.235 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:16.235 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:16.249 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:16.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:16.250 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:16.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:16.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:16.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:16.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:16.637 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:16.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:12:16.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:16 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:16.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:16.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:16 smithi107 ceph-mon[32381]: osdmap e440: 12 total, 3 up, 11 in 2024-08-06T18:12:16.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2794937720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:16.655 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:16 smithi107 ceph-mon[32381]: osdmap e441: 12 total, 3 up, 11 in 2024-08-06T18:12:16.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:16.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:16 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:16.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:16.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:16 smithi086 ceph-mon[29229]: osdmap e440: 12 total, 3 up, 11 in 2024-08-06T18:12:16.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2794937720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:16.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:16 smithi086 ceph-mon[29229]: osdmap e441: 12 total, 3 up, 11 in 2024-08-06T18:12:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:16 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1136273664' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:16 smithi117 ceph-mon[32835]: osdmap e440: 12 total, 3 up, 11 in 2024-08-06T18:12:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2794937720' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:16.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:16 smithi117 ceph-mon[32835]: osdmap e441: 12 total, 3 up, 11 in 2024-08-06T18:12:16.884 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:16.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:17.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:17.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:17.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:17.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:17.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:17.334 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:17.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:17.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:17.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:17.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: pgmap v1304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:12:17.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:17 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:12:17.760 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: pgmap v1304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:17.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:12:17.761 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:17 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:12:17.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:17.844 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: pgmap v1304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:17.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:17.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:12:17.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:17 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:12:17.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:18.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:18.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:18.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:18.462 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:18.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:18.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:18.749 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:18.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:18.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:18.874 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:18.875 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:18.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:19.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:19.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:19.175 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:19.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:12:19.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:19.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:12:19.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:19.423 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:19.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:19.597 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:19.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:19 smithi086 ceph-mon[29229]: pgmap v1306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:19.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:19.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:19.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:19.690 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:19 smithi107 ceph-mon[32381]: pgmap v1306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:19.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:12:19.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:19.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:19.864 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:19 smithi117 ceph-mon[32835]: pgmap v1306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:19.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:20.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:20.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:20.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:20.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:20.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:20.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:20.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:20.911 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:21.000 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:21.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:21.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:21.182 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:21.182 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:21.291 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:21.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:21.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:21.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:21.709 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:21.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:12:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:21 smithi107 ceph-mon[32381]: pgmap v1307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:21.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:21.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:21.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:21 smithi086 ceph-mon[29229]: pgmap v1307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:21 smithi117 ceph-mon[32835]: pgmap v1307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:21.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:22.020 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:22.058 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:22.058 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:22.059 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:22.070 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:22.070 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:22.070 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:22.071 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:22.071 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:22.071 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:22.089 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:22.090 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:22.090 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:22.090 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:22.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:22.091 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:22.091 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:22.092 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:22.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:22.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:22.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:22.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:22.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:22.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:22.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:22.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:22.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:22.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:22.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:22.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: osdmap e442: 12 total, 3 up, 11 in 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3181161478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:22.693 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:22.724 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:22.724 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: osdmap e442: 12 total, 3 up, 11 in 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3181161478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:22.725 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:22.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:22.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:22.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:12:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:12:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: osdmap e442: 12 total, 3 up, 11 in 2024-08-06T18:12:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3181161478' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:22.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:23.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:23.027 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:23.027 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:23.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:23.269 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:23.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:23.358 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:23.359 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:23.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:23.531 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:23.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:23.619 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:23 smithi086 ceph-mon[29229]: pgmap v1309: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:23.619 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:23 smithi117 ceph-mon[32835]: pgmap v1309: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:23.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:23.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:23.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:23.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:23 smithi107 ceph-mon[32381]: pgmap v1309: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:23.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:23.978 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:23.979 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:24.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:24.151 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:24.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:12:24.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:24.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:24.396 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:24.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:24.584 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:24.584 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:24.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:24.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:24.760 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:24 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:12:24.760 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:24 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:12:24.795 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:24 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:12:24.795 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:24 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:12:24.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:24.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:24 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:12:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:24 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:12:25.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:25.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:25.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:25.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:25.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:25.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:25.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:12:25.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:25.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:25.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:25.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:25.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:25.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:25.513 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:25.513 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":442,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:21.567699+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:25.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:25.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:25.586 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:12:25.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:25.758 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:25 smithi117 ceph-mon[32835]: pgmap v1310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:25.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:25.759 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3534436252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:25.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:25.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:25 smithi086 ceph-mon[29229]: pgmap v1310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:25.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:25.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3534436252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:25.960 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:25.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:25 smithi107 ceph-mon[32381]: pgmap v1310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:25.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3534436252' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:26.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:26.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:26.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:26.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:26.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:26.665 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:26.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:12:26.714 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:26.715 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:26.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:26 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:12:26.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:26.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:26 smithi086 ceph-mon[29229]: osdmap e443: 12 total, 3 up, 11 in 2024-08-06T18:12:26.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:26 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:12:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:26 smithi117 ceph-mon[32835]: osdmap e443: 12 total, 3 up, 11 in 2024-08-06T18:12:26.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:26.909 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:26 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:12:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/52949626' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:26 smithi107 ceph-mon[32381]: osdmap e443: 12 total, 3 up, 11 in 2024-08-06T18:12:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:27.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:27.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:27.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:27.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:27.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:27.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:27.712 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:27 smithi086 ceph-mon[29229]: pgmap v1312: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:27.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:27 smithi107 ceph-mon[32381]: pgmap v1312: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:27.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:27 smithi117 ceph-mon[32835]: pgmap v1312: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:27.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:28.044 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:28.045 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:28.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:28.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:28.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:28.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:28.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:12:28.474 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:28.484 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:28.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:28.721 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:28.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:28.757 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:12:28.758+0000 7f3adc8bc640 0 --2- 172.21.15.86:0/819401988 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ad81072b0 0x7f3ad81076f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:12:28.779 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:28.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:28.796 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:28.796 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:28.797 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:28.797 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:28.797 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:28.814 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:28.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:28.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:28.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:29.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:29.069 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:29.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:12:29.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:29.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:29.318 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:12:29.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:29.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:29.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:29.704 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:29 smithi117 ceph-mon[32835]: pgmap v1313: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:29.704 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1371769842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:29.704 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.704 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.713 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:29.713 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:29.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:29 smithi086 ceph-mon[29229]: pgmap v1313: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:29.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1371769842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:29.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:29.865 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:29.865 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:29.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:29 smithi107 ceph-mon[32381]: pgmap v1313: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:29.865 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1371769842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:29.866 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.866 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:29.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:30.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:30.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:30.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:30.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:30.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:30.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:30.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:30.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:30.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:30.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:30.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:30.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:30.920 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:30.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:31.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:31.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:31.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:31.238 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:31.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:31.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:31.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:31.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:31.594 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:31.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:12:31.616 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:12:31.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:31.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:31 smithi086 ceph-mon[29229]: pgmap v1314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:31.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:31 smithi117 ceph-mon[32835]: pgmap v1314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:31.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:31 smithi107 ceph-mon[32381]: pgmap v1314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:12:32.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:32.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:12:32.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:32.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:32.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:32.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:32.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:32.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:32 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:32.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:32.731 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:32 smithi086 ceph-mon[29229]: osdmap e444: 12 total, 3 up, 11 in 2024-08-06T18:12:32.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:32 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:32 smithi117 ceph-mon[32835]: osdmap e444: 12 total, 3 up, 11 in 2024-08-06T18:12:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:32 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/699600963' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:12:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:32 smithi107 ceph-mon[32381]: osdmap e444: 12 total, 3 up, 11 in 2024-08-06T18:12:32.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:33.027 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:33.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:33.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:33.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:33.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:33.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:33.675 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:33 smithi117 ceph-mon[32835]: pgmap v1316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:33.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:33.859 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:33.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:33 smithi086 ceph-mon[29229]: pgmap v1316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:33.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:33 smithi107 ceph-mon[32381]: pgmap v1316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:34.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:34.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:34.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:34.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:34.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:34.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:12:34.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:34.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:34.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:34.841 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:34.841 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:34.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:34.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:35.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:35.121 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:35.122 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":444,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:31.609359+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:35.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:35.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:35.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:35.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:35.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:35 smithi107 ceph-mon[32381]: pgmap v1317: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:35.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2172258855' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:35.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:35 smithi117 ceph-mon[32835]: pgmap v1317: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2172258855' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:35.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:35.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:35 smithi086 ceph-mon[29229]: pgmap v1317: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:35.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2172258855' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:35.931 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:35.931 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:36.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:36.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:36.225 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:36.226 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:36.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:36.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:36.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:36.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:36.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:36.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:36.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:36.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:37.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:37.155 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:37.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:37.377 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:37.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:37.415 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:37.415 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:37.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:37.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:37.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:37 smithi107 ceph-mon[32381]: pgmap v1318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:37.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:37 smithi107 ceph-mon[32381]: osdmap e445: 12 total, 3 up, 11 in 2024-08-06T18:12:37.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:37 smithi086 ceph-mon[29229]: pgmap v1318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:37.765 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:37 smithi086 ceph-mon[29229]: osdmap e445: 12 total, 3 up, 11 in 2024-08-06T18:12:37.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:37.829 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:37.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:12:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:37 smithi117 ceph-mon[32835]: pgmap v1318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:37 smithi117 ceph-mon[32835]: osdmap e445: 12 total, 3 up, 11 in 2024-08-06T18:12:37.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:38.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:38.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:38.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:38.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:38.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:38.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:38.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:38.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:38.758 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:38.758 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:38.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:38.875 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:38.875 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:38.983 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:38.983 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:38.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:38.995 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:39.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:39.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:39.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:39.015 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:39.015 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:39.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:39.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:39.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:39.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:39.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:39.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:39 smithi086 ceph-mon[29229]: pgmap v1320: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/49919754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.547 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:39.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:39.702 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:39 smithi117 ceph-mon[32835]: pgmap v1320: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:39.702 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/49919754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:39.702 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.702 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:39 smithi107 ceph-mon[32381]: pgmap v1320: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/49919754' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:39.812 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:39.815 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:39.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:40.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:40.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:40.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:40.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:40.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:40.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:40.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:40.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:40.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:40.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:40.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:40.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:40.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:40.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:40.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:41.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:41.042 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:41.042 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:41.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:41.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:41.308 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:41.309 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:41.327 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:41.327 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:41.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:41.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:41.574 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:41 smithi107 ceph-mon[32381]: pgmap v1321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:41.575 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:41.575 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:41.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:41.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:41 smithi086 ceph-mon[29229]: pgmap v1321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:41 smithi117 ceph-mon[32835]: pgmap v1321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:41.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:42.001 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:42.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:42.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:42.254 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:42.254 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:42.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:42.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:42.535 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:12:42.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:42.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:42.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:42.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:42.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:12:43.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:43.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:43.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:43.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:12:43.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:43.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:43.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:43.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:43.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:43.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:43 smithi086 ceph-mon[29229]: pgmap v1322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:43.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:43 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:43.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:43.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:43 smithi086 ceph-mon[29229]: osdmap e446: 12 total, 3 up, 11 in 2024-08-06T18:12:43.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:43.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:43 smithi117 ceph-mon[32835]: pgmap v1322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:43.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:43 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:43.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:43.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:43 smithi117 ceph-mon[32835]: osdmap e446: 12 total, 3 up, 11 in 2024-08-06T18:12:43.892 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:43.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:43 smithi107 ceph-mon[32381]: pgmap v1322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:43 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2884409084' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:12:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:43 smithi107 ceph-mon[32381]: osdmap e446: 12 total, 3 up, 11 in 2024-08-06T18:12:44.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:44.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:44.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:44.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:44.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:44.684 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:44.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:44.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:45.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:45.032 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:45.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:45.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:45.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:45.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:12:45.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:45.535 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:45.535 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":446,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:42.527466+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:45.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:45.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:45 smithi117 ceph-mon[32835]: pgmap v1324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:45.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:45.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:45 smithi086 ceph-mon[29229]: pgmap v1324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:45.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:45.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:45.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:45.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:45 smithi107 ceph-mon[32381]: pgmap v1324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:45.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:46.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:46.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:46.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:46.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:46.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:46.482 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:46.482 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:46.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:46.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:46.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:46.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3986505498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:46.818 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:46 smithi107 ceph-mon[32381]: osdmap e447: 12 total, 3 up, 11 in 2024-08-06T18:12:46.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3986505498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:46 smithi086 ceph-mon[29229]: osdmap e447: 12 total, 3 up, 11 in 2024-08-06T18:12:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3986505498' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:46 smithi117 ceph-mon[32835]: osdmap e447: 12 total, 3 up, 11 in 2024-08-06T18:12:46.921 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:46.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:12:46.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:47.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:47.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:47.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:47.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:47.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:47.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:47.546 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:47 smithi107 ceph-mon[32381]: pgmap v1325: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:47.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:47.651 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:47 smithi086 ceph-mon[29229]: pgmap v1325: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:47.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:47 smithi117 ceph-mon[32835]: pgmap v1325: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:47.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:47.968 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:47.968 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:48.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:48.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:12:48.067 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:12:48.067 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:12:48.083 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:12:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:12:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:12:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:12:48.084 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:12:48.101 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:12:48.102 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:12:48.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:12:48.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:12:48.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:12:48.102 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:12:48.103 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:12:48.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:48.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:48.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:48.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:48.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:48.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:48.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2140488923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:48.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.601 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:48.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2140488923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:48.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.731 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:48.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2140488923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:12:48.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:48.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:49.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:49.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:49.260 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:49.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:49.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:49.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:49.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:49 smithi107 ceph-mon[32381]: pgmap v1327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:49.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:49.768 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:12:49.768 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:12:49.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:49 smithi086 ceph-mon[29229]: pgmap v1327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:49.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:49.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:49.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:49 smithi117 ceph-mon[32835]: pgmap v1327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:12:49.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:50.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:50.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:50.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:50.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:50.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:50.558 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:12:50.636 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:50.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:50.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:50.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:50.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:50.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:12:50.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:51.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:51.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:51.232 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:51.232 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:51.322 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:51.322 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:51.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:51.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:51.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:51.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:51.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:51 smithi086 ceph-mon[29229]: pgmap v1328: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:51 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:51 smithi086 ceph-mon[29229]: osdmap e448: 12 total, 3 up, 11 in 2024-08-06T18:12:51.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:51 smithi086 ceph-mon[29229]: osdmap e449: 12 total, 3 up, 11 in 2024-08-06T18:12:51.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:51.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:12:51.851 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:12:51.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:51 smithi117 ceph-mon[32835]: pgmap v1328: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:51 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:51 smithi117 ceph-mon[32835]: osdmap e448: 12 total, 3 up, 11 in 2024-08-06T18:12:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:51 smithi117 ceph-mon[32835]: osdmap e449: 12 total, 3 up, 11 in 2024-08-06T18:12:51.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:51 smithi107 ceph-mon[32381]: pgmap v1328: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:51.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:51 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:12:51.891 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/854159688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:12:51.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:51 smithi107 ceph-mon[32381]: osdmap e448: 12 total, 3 up, 11 in 2024-08-06T18:12:51.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:51 smithi107 ceph-mon[32381]: osdmap e449: 12 total, 3 up, 11 in 2024-08-06T18:12:52.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:52.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:52.154 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:52.155 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:52.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:52.327 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:52.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:52.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:52.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:52.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:52.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:52.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:52.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:52.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:52.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:53.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:53.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:53.307 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:12:53.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:12:53.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:53.359 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:53.359 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:53.469 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:53.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:53.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:53.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:53.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:53 smithi107 ceph-mon[32381]: pgmap v1331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:53.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:53.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:53 smithi086 ceph-mon[29229]: pgmap v1331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:53 smithi117 ceph-mon[32835]: pgmap v1331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:53.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:54.001 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:54.009 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:54.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:54.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:54.361 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:54.361 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:54.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:54.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:54.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:54.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:54.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:12:54.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:55.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:55.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:55.265 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:12:55.265 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":449,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:12:51.518780+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:12:55.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:55.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:55.560 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:55.561 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:55.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:12:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:55 smithi117 ceph-mon[32835]: pgmap v1332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:55.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2393356451' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:55.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:55 smithi086 ceph-mon[29229]: pgmap v1332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:55.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2393356451' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:12:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:12:55.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:55 smithi107 ceph-mon[32381]: pgmap v1332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2393356451' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:12:56.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:56.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:56.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:56.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:56.404 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:56.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:56.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:56.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:56.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:56.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:56.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:57.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:57.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:57.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:57.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:12:57.525 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:12:57.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:57.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:57 smithi117 ceph-mon[32835]: pgmap v1333: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:57.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:12:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:57 smithi086 ceph-mon[29229]: pgmap v1333: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:57.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:12:57.898 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:57 smithi107 ceph-mon[32381]: pgmap v1333: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:58.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:58.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:12:58.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:58.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:58.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:12:58.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:12:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:12:58.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:58.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:58.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:12:58.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:12:58.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:58.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:12:59.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:59.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:12:59.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:59.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:12:59.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:12:59.579 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:12:59 smithi086 ceph-mon[29229]: pgmap v1334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:59.583 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:12:59.583 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:12:59.583 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:12:59.583 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:12:59.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:12:59.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:12:59 smithi107 ceph-mon[32381]: pgmap v1334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:59.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:12:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:12:59 smithi117 ceph-mon[32835]: pgmap v1334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:12:59.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:12:59.947 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:12:59.947 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:12:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:00.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:00.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:00.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:00.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:00.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:13:00.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:00.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:00.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:00.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:00.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:00.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:01.097 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:01.097 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:01.098 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:01.111 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:01.112 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:01.112 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:01.112 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:01.112 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:01.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:01.130 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:01.130 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:01.130 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:01.130 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:01.131 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:01.131 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:01.131 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:01.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:01.346 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:01.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:01.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:01.569 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:13:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:01 smithi086 ceph-mon[29229]: pgmap v1335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1341389297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:01.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:01.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:01.833 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:01 smithi117 ceph-mon[32835]: pgmap v1335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1341389297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:01.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:01 smithi107 ceph-mon[32381]: pgmap v1335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1341389297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:02.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:02.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:02.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:02.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:02.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:02 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:02.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:02 smithi117 ceph-mon[32835]: osdmap e450: 12 total, 3 up, 11 in 2024-08-06T18:13:02.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:02.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:02.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:02 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:02.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:02.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:02 smithi107 ceph-mon[32381]: osdmap e450: 12 total, 3 up, 11 in 2024-08-06T18:13:02.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:02.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:02.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:02 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3841648801' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:02.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:02 smithi086 ceph-mon[29229]: osdmap e450: 12 total, 3 up, 11 in 2024-08-06T18:13:02.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:03.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:03.138 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:03.138 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:03.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:03.272 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:03.272 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:03.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:03.455 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:03.455 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:03.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:03.581 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:03.581 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:03.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:03.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:03.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:03 smithi086 ceph-mon[29229]: pgmap v1337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:03 smithi117 ceph-mon[32835]: pgmap v1337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:03.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:03 smithi107 ceph-mon[32381]: pgmap v1337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:04.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:04.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:04.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:04.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:04.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:04.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:04.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:04.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:04.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:04.894 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:13:04.894 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":450,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:13:01.569735+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:13:04.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:05.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:05.099 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:05.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:05.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:05.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:05.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:05.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:05.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:05.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:05.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:05 smithi086 ceph-mon[29229]: pgmap v1338: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:05.683 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4260893940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:05.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:05 smithi117 ceph-mon[32835]: pgmap v1338: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4260893940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:05.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:05 smithi107 ceph-mon[32381]: pgmap v1338: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4260893940' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:06.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:13:06.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:06.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:06.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:06.431 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:13:06.431 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:13:06.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:06.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:06.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:06.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:06.876 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:06.876 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:06.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:07.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:13:07.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:07.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:07.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:07.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:07.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:07.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:13:07.690 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:07.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:07.790 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:07.791 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: pgmap v1339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: osdmap e451: 12 total, 3 up, 11 in 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:07.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:07 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: pgmap v1339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: osdmap e451: 12 total, 3 up, 11 in 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:07 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:07.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: pgmap v1339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: osdmap e451: 12 total, 3 up, 11 in 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:07.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:07 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:08.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:08.089 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:08.090 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:08.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:08.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:08.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:08.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:08.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:08.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:08.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:08.990 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:09.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:09.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:09.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:09.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:09.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:09.389 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:09.389 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:09.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:09.611 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:09.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:09.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:09 smithi086 ceph-mon[29229]: pgmap v1341: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:09.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:09.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:09.734 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:09.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:09.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:09 smithi117 ceph-mon[32835]: pgmap v1341: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:09.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:09.922 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:09 smithi107 ceph-mon[32381]: pgmap v1341: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:10.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:10.060 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:10.060 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:10.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:10.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:13:10.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:10.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:10.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:10.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:10.723 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:10.723 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:10.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:10.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:11.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:11.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:11.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:11.513 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:11.513 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:11.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:11.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:11.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:11.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:11 smithi086 ceph-mon[29229]: pgmap v1342: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:11 smithi117 ceph-mon[32835]: pgmap v1342: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:11.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:11.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:11 smithi107 ceph-mon[32381]: pgmap v1342: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:12.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:12.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:12.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:12.418 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:13:12.418 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:13:12.421 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:12.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:12.534 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:13:12.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:12.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:12.607 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:12.607 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:12.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:12.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:12.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:12.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:12.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:13.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:13.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:13.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:13.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:13.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:13.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:13.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:13.670 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:13 smithi117 ceph-mon[32835]: pgmap v1343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:13.670 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:13 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:13.670 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:13.670 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:13 smithi117 ceph-mon[32835]: osdmap e452: 12 total, 3 up, 11 in 2024-08-06T18:13:13.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:13.806 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:13 smithi107 ceph-mon[32381]: pgmap v1343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:13.806 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:13 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:13.806 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:13.806 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:13 smithi107 ceph-mon[32381]: osdmap e452: 12 total, 3 up, 11 in 2024-08-06T18:13:13.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:13 smithi086 ceph-mon[29229]: pgmap v1343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:13.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:13 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:13.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2521902889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:13.842 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:13 smithi086 ceph-mon[29229]: osdmap e452: 12 total, 3 up, 11 in 2024-08-06T18:13:13.843 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:13.843 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:13.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:14.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:14.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:14.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:14.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:14.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:14.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3757579994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:14.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:14.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3757579994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:14.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:14.885 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3757579994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:14.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:15.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:15.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:15.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:15.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:15.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:13:15.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:15.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:13:15.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:15.563 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:15.563 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:15.563 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:15.564 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:15.564 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:15.564 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:15.564 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:15.565 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:15.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:15.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:15 smithi107 ceph-mon[32381]: pgmap v1345: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:15.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:15.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:15 smithi086 ceph-mon[29229]: pgmap v1345: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:15.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:15 smithi117 ceph-mon[32835]: pgmap v1345: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:15.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:15.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:15.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:16.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:16.162 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:16.163 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:16.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:16.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:16.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:16.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:16.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:16.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:16.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:16.751 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:16.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:16.879 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:13:16.879 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":453,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:13:16.525404+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:47.041697+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:11:58.247862+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:13:17.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:17.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:17.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:17.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:17.526 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:17.526 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:17.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:17.568 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:17 smithi117 ceph-mon[32835]: pgmap v1346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:17.568 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:17 smithi117 ceph-mon[32835]: osdmap e453: 12 total, 3 up, 11 in 2024-08-06T18:13:17.568 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/614108118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:17.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:17.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:17 smithi107 ceph-mon[32381]: pgmap v1346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:17.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:17 smithi107 ceph-mon[32381]: osdmap e453: 12 total, 3 up, 11 in 2024-08-06T18:13:17.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/614108118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:17.719 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:17.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:17.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:17 smithi086 ceph-mon[29229]: pgmap v1346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:17.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:17 smithi086 ceph-mon[29229]: osdmap e453: 12 total, 3 up, 11 in 2024-08-06T18:13:17.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/614108118' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:17.828 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:17.828 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:17.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:18.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:18.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:13:18.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:18.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:18.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:18.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:18.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:18 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:13:18.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:18 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:13:18.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:18 smithi086 ceph-mon[29229]: osdmap e454: 12 total, 3 up, 11 in 2024-08-06T18:13:18.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:18.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:18 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:13:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:18 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:13:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:18 smithi117 ceph-mon[32835]: osdmap e454: 12 total, 3 up, 11 in 2024-08-06T18:13:18.944 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:18.944 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:18.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:18 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:13:18.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:18 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:13:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:18 smithi107 ceph-mon[32381]: osdmap e454: 12 total, 3 up, 11 in 2024-08-06T18:13:19.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:19.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:19.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:19.298 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:19.298 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:19.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:19.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:19.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:19.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:19.563 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:19 smithi107 ceph-mon[32381]: pgmap v1349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:19.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:19.676 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:19 smithi086 ceph-mon[29229]: pgmap v1349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:19.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:19.834 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:19 smithi117 ceph-mon[32835]: pgmap v1349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:19.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:19.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:19.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:20.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:20.214 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:20.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:20.451 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:20.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:20.553 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:13:20.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:20.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:20.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:13:20.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:20.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:20.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:20.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:20.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:20.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:20.974 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:21.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:21.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:21.257 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:21.257 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:21.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:21.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:21.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:21 smithi117 ceph-mon[32835]: pgmap v1350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:21 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:21.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:21 smithi117 ceph-mon[32835]: osdmap e455: 12 total, 3 up, 11 in 2024-08-06T18:13:21.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:21.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:21.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:21.705 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:21 smithi107 ceph-mon[32381]: pgmap v1350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:21.705 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:21 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:21.705 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:21.705 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:21 smithi107 ceph-mon[32381]: osdmap e455: 12 total, 3 up, 11 in 2024-08-06T18:13:21.744 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:13:21.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:21.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:21 smithi086 ceph-mon[29229]: pgmap v1350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:21 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1754955922' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:21 smithi086 ceph-mon[29229]: osdmap e455: 12 total, 3 up, 11 in 2024-08-06T18:13:21.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:22.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:22.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:22.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:22.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:22.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:22.421 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:22.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:22.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:22.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:22.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:13:22.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:13:22.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:22.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:23.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:23.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:23.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:23.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:13:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:23.305 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:23.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:23.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:23.565 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:23.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:23 smithi086 ceph-mon[29229]: pgmap v1352: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:23.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:23.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:23.841 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:23 smithi107 ceph-mon[32381]: pgmap v1352: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:23 smithi117 ceph-mon[32835]: pgmap v1352: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:23.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:24.006 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:24.006 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:24.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:24.131 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:24.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:24.140 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:24.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:24.142 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:24.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:24.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:24.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:24.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:24.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:24.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:24.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3749192165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:24.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:24.776 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:24.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:24.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:24.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:24.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3749192165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:24.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:24.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3749192165' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:24.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:25.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:25.035 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:25.035 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:25.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:25.051 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:25.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:25.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:25.338 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:25.339 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:25.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:25.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:25.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:25 smithi086 ceph-mon[29229]: pgmap v1353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:25.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:25.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:25.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:25.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:25 smithi117 ceph-mon[32835]: pgmap v1353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:25.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:25.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:25 smithi107 ceph-mon[32381]: pgmap v1353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:26.009 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:26.009 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:26.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:26.140 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:26.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:26.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:26.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:26.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:26.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:26.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:26.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:26.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:27.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:27.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:13:27.063 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:27.063 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:27.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:27.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:27.325 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:27.326 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:27.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:27.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:27.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:27.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:27.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:27 smithi086 ceph-mon[29229]: pgmap v1354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:27.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:27.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:27.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:27 smithi117 ceph-mon[32835]: pgmap v1354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:27 smithi107 ceph-mon[32381]: pgmap v1354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:27.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:27.981 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:27.981 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:28.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:28.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:28.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:28.481 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:13:28.481 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":456,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:13:27.563162+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:13:28.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:28.545 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:28.545 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:28.578 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:13:28.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:28.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:28.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:28.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:28.796 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:28.796 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:28.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:28 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:13:28.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:28 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:13:28.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:28 smithi086 ceph-mon[29229]: osdmap e456: 12 total, 3 up, 11 in 2024-08-06T18:13:28.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:13:28.797 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2747518542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:28 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:13:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:28 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:13:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:28 smithi117 ceph-mon[32835]: osdmap e456: 12 total, 3 up, 11 in 2024-08-06T18:13:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:13:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2747518542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:28.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:28 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:13:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:28 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:13:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:28 smithi107 ceph-mon[32381]: osdmap e456: 12 total, 3 up, 11 in 2024-08-06T18:13:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:13:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2747518542' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:29.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:29.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:29.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:29.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:29.461 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:29.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:29.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:29.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:29.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:29 smithi107 ceph-mon[32381]: pgmap v1356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:29 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:13:29.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:29 smithi107 ceph-mon[32381]: osdmap e457: 12 total, 3 up, 11 in 2024-08-06T18:13:29.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:29 smithi086 ceph-mon[29229]: pgmap v1356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:29.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:29 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:29.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:13:29.728 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:29 smithi086 ceph-mon[29229]: osdmap e457: 12 total, 3 up, 11 in 2024-08-06T18:13:29.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:29.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:29 smithi117 ceph-mon[32835]: pgmap v1356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:29.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:29 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:29.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4252363114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:13:29.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:29 smithi117 ceph-mon[32835]: osdmap e457: 12 total, 3 up, 11 in 2024-08-06T18:13:29.913 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:29.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:30.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:30.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:30.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:30.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:30.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:30.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:30.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:30.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:30.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:31.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:31.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:31.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:31.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:31.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:13:31.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:31.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:31.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:31.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:31 smithi086 ceph-mon[29229]: pgmap v1358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:31 smithi117 ceph-mon[32835]: pgmap v1358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:31 smithi107 ceph-mon[32381]: pgmap v1358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:31.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:32.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:32.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:32.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:32.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:32.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:32.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:32.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:32.694 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:13:32.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:32.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:33.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:33.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:33.197 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:33.197 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:33.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:33.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:33.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:13:33.520 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:13:33.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:13:33.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:33.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:33.751 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:33 smithi086 ceph-mon[29229]: pgmap v1359: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:33.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:33 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:13:33.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:33 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:13:33.780 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:33.781 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:33.781 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:33.795 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:33.803 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:33.804 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:33.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:33.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:33.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:33.813 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:33.826 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:33.826 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:33.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:33.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:33.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:33.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:33.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:33.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:33.867 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:33.867 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:33.867 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:33 smithi107 ceph-mon[32381]: pgmap v1359: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:33.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:33 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:13:33.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:33 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:13:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:33 smithi117 ceph-mon[32835]: pgmap v1359: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:33 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:13:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:33 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:13:33.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:34.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:34.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:34.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:34.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:34.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:34.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:34.550 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:34.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:34.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:34.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:34.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/631706831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:34.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:34.794 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:34.794 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:34.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/631706831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:34.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.795 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/631706831' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:34.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:35.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:35.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:35.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:35.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:35.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:35.430 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:35.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:35.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:35.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:35.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:35.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:35.726 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:35 smithi117 ceph-mon[32835]: pgmap v1360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:35.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:35.862 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:35.862 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:35 smithi107 ceph-mon[32381]: pgmap v1360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:35.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:35.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:35 smithi086 ceph-mon[29229]: pgmap v1360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:35.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:36.005 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:36.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:36.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:36.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:36.500 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:36.501 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:36.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:36.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:13:36.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:36.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:36.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:37.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:37.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:37.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:37.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:37.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:37.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:37.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:37.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:37.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:37 smithi086 ceph-mon[29229]: pgmap v1361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:37.872 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:37 smithi107 ceph-mon[32381]: pgmap v1361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:37 smithi117 ceph-mon[32835]: pgmap v1361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:37.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:38.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:38.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:38.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:38.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:38.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:38.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:38.614 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:13:38.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:38.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:38.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:38.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:38.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:39.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:39.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:39.205 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:39.206 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:39.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:39.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:39.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:39.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: pgmap v1362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:39.764 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:39 smithi086 ceph-mon[29229]: osdmap e458: 12 total, 3 up, 11 in 2024-08-06T18:13:39.788 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:13:39.788 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":458,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:13:38.608794+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:12:21.084597+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:13:39.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: pgmap v1362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:39 smithi117 ceph-mon[32835]: osdmap e458: 12 total, 3 up, 11 in 2024-08-06T18:13:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: pgmap v1362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:13:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:13:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:13:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/103294293' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:39 smithi107 ceph-mon[32381]: osdmap e458: 12 total, 3 up, 11 in 2024-08-06T18:13:39.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:40.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:40.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:40.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:40.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:40.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:40.679 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/238134096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:40.679 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:40.679 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:40.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/238134096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:40.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/238134096' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:40.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:40.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:40.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:41.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:41.078 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:41.078 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:41.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:41.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:41.344 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:41.344 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:41.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:41.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:41.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:41.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:41 smithi086 ceph-mon[29229]: pgmap v1364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:41 smithi086 ceph-mon[29229]: osdmap e459: 12 total, 3 up, 11 in 2024-08-06T18:13:41.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:41.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:41 smithi117 ceph-mon[32835]: pgmap v1364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:41 smithi117 ceph-mon[32835]: osdmap e459: 12 total, 3 up, 11 in 2024-08-06T18:13:41.878 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:41 smithi107 ceph-mon[32381]: pgmap v1364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:41.878 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:41 smithi107 ceph-mon[32381]: osdmap e459: 12 total, 3 up, 11 in 2024-08-06T18:13:41.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:42.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:42.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:42.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:42.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:42.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:42.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:42.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:13:42.615 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:42.615 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:42.616 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:42.620 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:42.620 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:42.634 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:42.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:42.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:42.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:42.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:42.652 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:42.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:42.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:42.653 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:42.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:42.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:42.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1844544891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1844544891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:42.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:42.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1844544891' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:43.032 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:13:43.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:43.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:43.323 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:43.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:43.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:43.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:43.741 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:13:43.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:13:43.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:43 smithi086 ceph-mon[29229]: pgmap v1366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:43 smithi117 ceph-mon[32835]: pgmap v1366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:43 smithi107 ceph-mon[32381]: pgmap v1366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:43.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:44.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:44.190 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:44.190 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:44.190 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:44.191 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:44.224 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:44.225 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:44.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:44.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:44.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:13:44.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:44.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:44.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:44.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:44.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:44.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:45.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:45.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:45.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:45.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:45.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:45.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:45.346 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:45.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:45.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:45.613 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:45.613 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:45.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:45.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:45.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:45 smithi086 ceph-mon[29229]: pgmap v1367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:45.882 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:45 smithi117 ceph-mon[32835]: pgmap v1367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:45.906 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:45.906 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:45 smithi107 ceph-mon[32381]: pgmap v1367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:45.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:45.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:45.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:46.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:46.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:13:46.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:46.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:46.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:46.632 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:13:46.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:46.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:46.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:46.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:46.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:46.816 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:46.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:47.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:13:47.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:47.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:47.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:47.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:47.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:47.775 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:47.775 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:47.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:47.776 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:47.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:47 smithi086 ceph-mon[29229]: pgmap v1368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:47.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:47 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:47.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:47.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:47 smithi086 ceph-mon[29229]: osdmap e460: 12 total, 3 up, 11 in 2024-08-06T18:13:47.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:47.878 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:47 smithi117 ceph-mon[32835]: pgmap v1368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:47.878 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:47 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:47.878 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:47.878 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:47 smithi117 ceph-mon[32835]: osdmap e460: 12 total, 3 up, 11 in 2024-08-06T18:13:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:47 smithi107 ceph-mon[32381]: pgmap v1368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:47 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3030561679' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:13:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:47 smithi107 ceph-mon[32381]: osdmap e460: 12 total, 3 up, 11 in 2024-08-06T18:13:47.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:48.084 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:48.084 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:48.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:48.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:48.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:48.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:48.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:48.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:48.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:48.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:49.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:49.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:49.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:49.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:49.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:49.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:49.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:49.461 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:49.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:49.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:49.741 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:49 smithi107 ceph-mon[32381]: pgmap v1370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:49.741 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:49.741 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:49.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:49.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:49 smithi086 ceph-mon[29229]: pgmap v1370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:49.883 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:49 smithi117 ceph-mon[32835]: pgmap v1370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:49.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:50.061 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:13:50.061 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:13:50.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:50.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:50.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:50.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:50.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:50.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:13:50.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:50.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:50.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:50.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:50.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:50.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:51.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:13:51.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:51.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:51.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:51.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:51.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:51.637 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:51.637 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:51.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:51.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:51 smithi086 ceph-mon[29229]: pgmap v1371: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:51.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:51 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:13:51.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:51 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:13:51.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:51 smithi086 ceph-mon[29229]: osdmap e461: 12 total, 3 up, 11 in 2024-08-06T18:13:51.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:51 smithi107 ceph-mon[32381]: pgmap v1371: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:51 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:13:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:51 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:13:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:51 smithi107 ceph-mon[32381]: osdmap e461: 12 total, 3 up, 11 in 2024-08-06T18:13:52.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:52.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:51 smithi117 ceph-mon[32835]: pgmap v1371: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:51 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:13:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:51 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:13:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:51 smithi117 ceph-mon[32835]: osdmap e461: 12 total, 3 up, 11 in 2024-08-06T18:13:52.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:52.159 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:13:52.159 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:13:52.159 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:13:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:13:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:13:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:13:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:13:52.160 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:13:52.174 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:13:52.175 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:13:52.175 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:13:52.175 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:13:52.175 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:13:52.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:13:52.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:13:52.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:52.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:52.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:52.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:52.708 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2263999602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:52.708 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:52.709 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:52.709 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:52.786 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:13:52.786 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:13:52.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:52.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2263999602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:52.973 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2263999602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:13:52.973 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.003 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:53.003 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:53.040 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:53.047 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:53.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:53.207 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:13:53.207 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":461,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:13:50.633403+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:13:53.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:53.275 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:53.275 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:53.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:53.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:53.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:53.643 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:53.643 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:53.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:53 smithi086 ceph-mon[29229]: pgmap v1373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1099552384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:53.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:53.915 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:53 smithi117 ceph-mon[32835]: pgmap v1373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:53.916 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.916 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.916 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1099552384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:53 smithi107 ceph-mon[32381]: pgmap v1373: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:13:53.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1099552384' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:13:54.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:54.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:54.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:54.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:54.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:54.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:54.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:54.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:54.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:54.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:54.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:13:54.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:54.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:55.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:55.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:55.252 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:55.252 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:55.253 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:55.253 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:55.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:55.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:55.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:55.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:55.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:13:55.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:55 smithi086 ceph-mon[29229]: pgmap v1374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:13:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:13:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:13:55.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:55.924 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:55 smithi107 ceph-mon[32381]: pgmap v1374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:56.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:55 smithi117 ceph-mon[32835]: pgmap v1374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:56.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:56.193 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:56.193 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:56.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:56.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:56.465 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:56.465 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:56.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:13:56.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:56.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:13:56.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:13:56.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:13:56.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:56.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:56 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:56.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:13:56.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:13:56.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:13:56.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:56.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:56 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:56 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:56.971 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:56.972 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:57.071 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:13:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:13:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:13:57.071 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:13:57.071 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:13:57.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:57.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:57.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:57.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:57.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:57.661 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:13:57.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:13:57.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:58.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:58.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:58.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:58.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.028 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:58.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:13:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:13:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:13:58.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:58.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:58.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:58.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:58.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:13:58.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:13:58.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:58.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:58 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:58.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:58.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:58 smithi086 ceph-mon[29229]: osdmap e462: 12 total, 3 up, 11 in 2024-08-06T18:13:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:58 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:58 smithi117 ceph-mon[32835]: osdmap e462: 12 total, 3 up, 11 in 2024-08-06T18:13:58.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:13:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:58 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:13:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2549877170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:13:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:58 smithi107 ceph-mon[32381]: osdmap e462: 12 total, 3 up, 11 in 2024-08-06T18:13:59.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:13:59.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:13:59.212 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:13:59.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:13:59.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:13:59.468 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:13:59.468 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:13:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:13:59.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:13:59.664 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:13:59.664 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:13:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:13:59.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:13:59.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:13:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:13:59 smithi086 ceph-mon[29229]: pgmap v1377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:13:59 smithi117 ceph-mon[32835]: pgmap v1377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:13:59 smithi107 ceph-mon[32381]: pgmap v1377: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:13:59.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:00.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:00.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:00.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:00.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:00.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:00.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:00.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:14:00.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:00.803 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:00.803 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:00.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:00.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:00.821 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:00.822 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:00.822 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:00.822 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:00.822 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:00.831 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:00.844 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:00.844 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:00.844 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:00.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:00.937 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:00.938 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:00.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:01.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:01.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:01.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:01.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:01.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:01.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:01.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:01.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:01.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:01 smithi086 ceph-mon[29229]: pgmap v1378: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:01.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2446747008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:01.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:01.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:01.757 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:01 smithi086 ceph-mon[29229]: osdmap e463: 12 total, 3 up, 11 in 2024-08-06T18:14:01.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:01.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:01 smithi107 ceph-mon[32381]: pgmap v1378: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2446747008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:01 smithi107 ceph-mon[32381]: osdmap e463: 12 total, 3 up, 11 in 2024-08-06T18:14:01.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:14:02.010 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:14:02.011 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:02.011 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:02.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:01 smithi117 ceph-mon[32835]: pgmap v1378: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:02.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2446747008' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:02.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:02.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:02.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:01 smithi117 ceph-mon[32835]: osdmap e463: 12 total, 3 up, 11 in 2024-08-06T18:14:02.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:02.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:02.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:02.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:02.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:02.697 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:02.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:02.833 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:02.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:02.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:03.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:03.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:03.236 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:03.236 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:03.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:03.399 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:03.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:03.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:03.604 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:03.605 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:03.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:03.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:03.779 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:03 smithi107 ceph-mon[32381]: pgmap v1380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:03.779 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:03.779 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:03.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:03 smithi086 ceph-mon[29229]: pgmap v1380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:03.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:03.914 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:03 smithi117 ceph-mon[32835]: pgmap v1380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:03.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:14:03.944 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:04.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:04.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:04.181 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:04.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:04.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:04.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:04.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:04.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:04.864 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:04.864 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:04.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:05.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:05.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:05.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:05.393 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:05.393 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:05.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:14:05.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:05.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:05.678 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:05.679 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:05.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:05.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:05.912 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:05.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:05 smithi086 ceph-mon[29229]: pgmap v1381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:05 smithi107 ceph-mon[32381]: pgmap v1381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:06.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:06.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:05 smithi117 ceph-mon[32835]: pgmap v1381: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:06.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:06.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:06.342 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:06.343 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":463,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:01.536260+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:06.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:06.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:06.580 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:06.580 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:06.581 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:06.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:14:06.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:06.831 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:06.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:06.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2716091377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:06.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2716091377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:06.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:06.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2716091377' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:07.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:07.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:07.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:07.459 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:07.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:07.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:07.686 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:14:07.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:07.784 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:07 smithi107 ceph-mon[32381]: pgmap v1382: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:07.784 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:07.862 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:07 smithi086 ceph-mon[29229]: pgmap v1382: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:07.862 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:07.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:07.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:07 smithi117 ceph-mon[32835]: pgmap v1382: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 323 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:07.920 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:08.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:08.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:08.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:08.308 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:08.309 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:08.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:08.509 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:08.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:08.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:08.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:08.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:08.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:08 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:08.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:08 smithi086 ceph-mon[29229]: osdmap e464: 12 total, 3 up, 11 in 2024-08-06T18:14:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:08 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:08 smithi107 ceph-mon[32381]: osdmap e464: 12 total, 3 up, 11 in 2024-08-06T18:14:08.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:09.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:09.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:08 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:09.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4034430637' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:09.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:08 smithi117 ceph-mon[32835]: osdmap e464: 12 total, 3 up, 11 in 2024-08-06T18:14:09.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:09.270 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:09.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:14:09.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:09.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:09.496 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:09.496 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:09.520 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:09.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:09.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: pgmap v1384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=133278866,config=507150784,health=1738671092,kv=1723772696}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018673312}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33561260}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=496694879}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584493131}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079339363}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545945991}) 2024-08-06T18:14:09.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=143673374}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250424993}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397809201}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445992205}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768976565}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365302403}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454032346}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010649496}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707085718}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371697701}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069847528}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1953410058}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54130760}) 2024-08-06T18:14:09.772 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3640223634}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154568613}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085106761}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1217366109}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191205366}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790789070}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566163222}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704252308}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585931248}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2287215544}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934264279}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049794665}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561969110}) 2024-08-06T18:14:09.773 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448455651}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032101196}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063142178}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118058655}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238656069}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20824728}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754071316}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359855786}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089928465}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399666712}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771368781}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315181219}) 2024-08-06T18:14:09.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671842768}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1080653743}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350363760}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146143977}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827236246}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741241797}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=5} crc {logm=3266473900,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4094470715,mon_config_key=423435}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3337159126}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2036287979}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=74,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=8} crc {mon_config_key=1082575965,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3585172108,osdmap=424515022}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2452764823}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1184525343}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=502826293}) 2024-08-06T18:14:09.775 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1544915485}) 2024-08-06T18:14:09.776 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3179189703}) 2024-08-06T18:14:09.776 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2014523694}) 2024-08-06T18:14:09.776 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=54} crc {osdmap=2154621207}) 2024-08-06T18:14:09.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: pgmap v1384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:09.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=133278866,config=507150784,health=1738671092,kv=1723772696}) 2024-08-06T18:14:09.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018673312}) 2024-08-06T18:14:09.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33561260}) 2024-08-06T18:14:09.782 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=496694879}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584493131}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079339363}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545945991}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=143673374}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250424993}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397809201}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445992205}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768976565}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365302403}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454032346}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010649496}) 2024-08-06T18:14:09.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707085718}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371697701}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069847528}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1953410058}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54130760}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3640223634}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154568613}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085106761}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1217366109}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191205366}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790789070}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566163222}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704252308}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585931248}) 2024-08-06T18:14:09.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2287215544}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934264279}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049794665}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561969110}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448455651}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032101196}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063142178}) 2024-08-06T18:14:09.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118058655}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238656069}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20824728}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754071316}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359855786}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089928465}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399666712}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771368781}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315181219}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671842768}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1080653743}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350363760}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146143977}) 2024-08-06T18:14:09.786 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827236246}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741241797}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=5} crc {logm=3266473900,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4094470715,mon_config_key=423435}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3337159126}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2036287979}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=74,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=8} crc {mon_config_key=1082575965,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3585172108,osdmap=424515022}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2452764823}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1184525343}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=502826293}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1544915485}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3179189703}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2014523694}) 2024-08-06T18:14:09.787 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=54} crc {osdmap=2154621207}) 2024-08-06T18:14:09.890 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:09.890 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:09.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:09.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:09.903 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:09.904 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:09.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:09.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:09.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:09.922 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:09.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:10.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: pgmap v1384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=33,config=2,health=12,kv=53} crc {auth=133278866,config=507150784,health=1738671092,kv=1723772696}) 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3018673312}) 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=33561260}) 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=496694879}) 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=584493131}) 2024-08-06T18:14:10.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1079339363}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1545945991}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=143673374}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3250424993}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1397809201}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3445992205}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3768976565}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=365302403}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3454032346}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010649496}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707085718}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2371697701}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2069847528}) 2024-08-06T18:14:10.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1953410058}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=54130760}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3640223634}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4154568613}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1085106761}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1217366109}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191205366}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2790789070}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2566163222}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3704252308}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585931248}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2287215544}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=934264279}) 2024-08-06T18:14:10.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4049794665}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1561969110}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3448455651}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032101196}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1063142178}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2118058655}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4238656069}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=20824728}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1754071316}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2359855786}) 2024-08-06T18:14:10.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4089928465}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1399666712}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2771368781}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1315181219}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2671842768}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1080653743}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=350363760}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=146143977}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3827236246}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2741241797}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=61,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=5} crc {logm=3266473900,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4094470715,mon_config_key=423435}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3337159126}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2036287979}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=74,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=8} crc {mon_config_key=1082575965,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=3585172108,osdmap=424515022}) 2024-08-06T18:14:10.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2452764823}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1184525343}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=502826293}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1544915485}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3179189703}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2014523694}) 2024-08-06T18:14:10.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=54} crc {osdmap=2154621207}) 2024-08-06T18:14:10.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:10.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:10.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:10.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:10.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:10.431 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:10.431 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:10.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:10.572 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:10.572 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:10.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:10.641 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:10.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:10.709 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:10.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:10.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1318738110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:10.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:10.774 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:10.775 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:10.775 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:10.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:11.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:11.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1318738110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:11.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:11.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:11.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1318738110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:11.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:11.168 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:11.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:11.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:14:11.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:11.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:11.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:11.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:11 smithi117 ceph-mon[32835]: pgmap v1385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:11.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:11 smithi117 ceph-mon[32835]: osdmap e465: 12 total, 3 up, 11 in 2024-08-06T18:14:11.915 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:11.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:14:11.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:11.953 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.953 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:11 smithi086 ceph-mon[29229]: pgmap v1385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:11.953 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:11 smithi086 ceph-mon[29229]: osdmap e465: 12 total, 3 up, 11 in 2024-08-06T18:14:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:11 smithi107 ceph-mon[32381]: pgmap v1385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:11 smithi107 ceph-mon[32381]: osdmap e465: 12 total, 3 up, 11 in 2024-08-06T18:14:12.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:12.166 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:12.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:12.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:12.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:12.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:12.348 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:12.349 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:12.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:12.501 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:12.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:14:12.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:12.635 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":465,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:11.539132+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:12.653 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:12.653 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:12.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:12.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3489926907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:12.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:12.772 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:12.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:12.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3489926907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:13.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:13.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3489926907' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:13.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:13.297 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:13.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:13.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:13.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:13.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:13.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:13.717 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:13.718 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:13.747 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:13.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:13.850 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:13 smithi107 ceph-mon[32381]: pgmap v1387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:13.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:13.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:13 smithi086 ceph-mon[29229]: pgmap v1387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:14.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:13 smithi117 ceph-mon[32835]: pgmap v1387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:14.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:14.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:14.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:14.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:14:14.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:14.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:14.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:14.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:14.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:14.994 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:15.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:15.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:15.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:15.416 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:15.416 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:15.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:15.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:15.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:15.721 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:15.721 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:15.721 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:14:15.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:15.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:15.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:15 smithi107 ceph-mon[32381]: pgmap v1388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:15.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:15.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:15 smithi086 ceph-mon[29229]: pgmap v1388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:15.992 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:15.993 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:15.993 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:16.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:15 smithi117 ceph-mon[32835]: pgmap v1388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:16.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:16.294 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:16.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:16.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:16.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:16.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:16.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:16.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:16.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:16 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:16 smithi117 ceph-mon[32835]: osdmap e466: 12 total, 3 up, 11 in 2024-08-06T18:14:16.810 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:16 smithi117 ceph-mon[32835]: osdmap e467: 12 total, 3 up, 11 in 2024-08-06T18:14:16.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:16.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:16 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:16.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:16.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:16 smithi107 ceph-mon[32381]: osdmap e466: 12 total, 3 up, 11 in 2024-08-06T18:14:16.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:16 smithi107 ceph-mon[32381]: osdmap e467: 12 total, 3 up, 11 in 2024-08-06T18:14:16.989 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:16.989 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:17.015 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:17.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:14:17.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:16 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:17.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3930444003' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:17.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:16 smithi086 ceph-mon[29229]: osdmap e466: 12 total, 3 up, 11 in 2024-08-06T18:14:17.027 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:16 smithi086 ceph-mon[29229]: osdmap e467: 12 total, 3 up, 11 in 2024-08-06T18:14:17.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:17.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:17.274 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:17.279 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:17.279 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:17.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:17.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:17.298 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:17.298 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:17.299 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:17.304 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:17.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:17.319 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:17.319 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:17.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:17.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:17.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:17.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:17.320 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:17.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:17.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:17.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:17.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:17.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:17.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:17.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:17.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:17.987 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:17.987 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:17.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:17 smithi086 ceph-mon[29229]: pgmap v1390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:17.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3195102120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:17.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:18.121 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:17 smithi117 ceph-mon[32835]: pgmap v1390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:18.121 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3195102120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:18.121 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.136 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:17 smithi107 ceph-mon[32381]: pgmap v1390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:18.136 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3195102120' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:18.136 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:18.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:18.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:18.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:18.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:18.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:18.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:18.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:14:18.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:18.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.959 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:18.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:18.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:19.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:19.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:19.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:19.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:19.528 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:19.528 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:19.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:19.615 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:19.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:14:19.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:19.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:19.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:19.787 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:19.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:19.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:19 smithi086 ceph-mon[29229]: pgmap v1392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:19.914 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:19.920 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:19.920 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":467,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:16.540750+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:17.250234+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:19 smithi107 ceph-mon[32381]: pgmap v1392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:19.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:19.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:20.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:20.121 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:19 smithi117 ceph-mon[32835]: pgmap v1392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:20.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:20.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:20.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:20.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:20.654 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:20.654 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:20.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:20.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2405243921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:20.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2405243921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:21.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:21.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2405243921' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:21.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:21.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:21.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:21.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:21.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:21.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:21.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:21.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:14:21.545 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:14:21.545 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:14:21.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:21.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:21.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:21 smithi086 ceph-mon[29229]: pgmap v1393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:21.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:21.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:21.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:21 smithi107 ceph-mon[32381]: pgmap v1393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:21.965 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:21.965 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:22.092 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:22.092 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:22.093 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:22.093 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:22.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:22.121 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:21 smithi117 ceph-mon[32835]: pgmap v1393: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:22.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:22.235 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:22.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:14:22.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:22.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:22.550 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:22.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:22.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:22.753 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:14:22.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:22.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:22.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:22.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:23.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:23.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:23.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:23.211 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:23.211 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:23.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:23.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:23.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:23.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:23.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:23.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:23 smithi117 ceph-mon[32835]: pgmap v1394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:23.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:23 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:23.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:23.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:23 smithi117 ceph-mon[32835]: osdmap e468: 12 total, 3 up, 11 in 2024-08-06T18:14:23.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:23 smithi086 ceph-mon[29229]: pgmap v1394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:23.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:23 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:23 smithi086 ceph-mon[29229]: osdmap e468: 12 total, 3 up, 11 in 2024-08-06T18:14:23.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:23 smithi107 ceph-mon[32381]: pgmap v1394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:23 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/233584940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:23 smithi107 ceph-mon[32381]: osdmap e468: 12 total, 3 up, 11 in 2024-08-06T18:14:24.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:24.224 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:24.225 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:24.241 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:14:24.241 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:14:24.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:24.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:24.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:24.643 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:24.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:24.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:24.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2481354925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:24.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:14:24.830 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:24.831 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:24.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:24.847 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:24.848 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:24.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:24.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:24.848 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:24.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:24.867 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:24.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:24.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:24.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:24.868 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:24.868 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:24.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:25.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:25.040 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2481354925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2481354925' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:25.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:25.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:25.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:25.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:25.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:25.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:25.579 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:25.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:14:25.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:25.719 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:25.719 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:25.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:25.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:25.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:25 smithi086 ceph-mon[29229]: pgmap v1396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:25.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:25.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:25.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:25.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:25 smithi107 ceph-mon[32381]: pgmap v1396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:25.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:25.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:25.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:26.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:25 smithi117 ceph-mon[32835]: pgmap v1396: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:26.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:26.275 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:26.275 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:26.328 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:26.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:26.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:26.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:26.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:26.759 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:26.760 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:26.805 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:14:26.806 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:14:26.806 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:14:26.806 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:14:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:387,from:0,reservation_nonce:0)", 2024-08-06T18:14:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:14:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.25363044699998, 2024-08-06T18:14:26.807 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00028996399999999998, 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.808 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:14:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:14:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.809 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:14:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553020+0000", 2024-08-06T18:14:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.810 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:14:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553021+0000", 2024-08-06T18:14:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.345e-06 2024-08-06T18:14:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.811 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:14:26.812 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553027+0000", 2024-08-06T18:14:26.812 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1750000000000002e-06 2024-08-06T18:14:26.812 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.812 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553041+0000", 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3604000000000001e-05 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.813 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:14:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553068+0000", 2024-08-06T18:14:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.7019999999999999e-05 2024-08-06T18:14:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.814 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:14:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553168+0000", 2024-08-06T18:14:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000100074 2024-08-06T18:14:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.815 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553185+0000", 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7234000000000001e-05 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:14:26.816 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553310+0000", 2024-08-06T18:14:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012451199999999999 2024-08-06T18:14:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:14:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.817 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:14:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:14:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 340.25279341300001, 2024-08-06T18:14:26.818 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.5891999999999996e-05, 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.819 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:14:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:14:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.820 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:14:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:14:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.821 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:14:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553857+0000", 2024-08-06T18:14:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7400000000000003e-07 2024-08-06T18:14:26.822 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:14:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553861+0000", 2024-08-06T18:14:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.3569999999999998e-06 2024-08-06T18:14:26.823 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:14:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553874+0000", 2024-08-06T18:14:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3671e-05 2024-08-06T18:14:26.824 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553880+0000", 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.4399999999999996e-06 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.825 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:14:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553923+0000", 2024-08-06T18:14:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3322000000000003e-05 2024-08-06T18:14:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.826 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553930+0000", 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4479999999999999e-06 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.827 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:14:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:08:46.553953+0000", 2024-08-06T18:14:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.3079999999999999e-05 2024-08-06T18:14:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:14:26.828 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:423,from:0,reservation_nonce:0)", 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 187.787604286, 2024-08-06T18:14:26.829 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030840499999999999, 2024-08-06T18:14:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:14:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:14:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:14:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:14:26.830 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.831 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:14:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019046+0000", 2024-08-06T18:14:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.832 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:14:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019047+0000", 2024-08-06T18:14:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.44e-06 2024-08-06T18:14:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.833 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:14:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019054+0000", 2024-08-06T18:14:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.9269999999999999e-06 2024-08-06T18:14:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.834 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019070+0000", 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5523e-05 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:14:26.835 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019096+0000", 2024-08-06T18:14:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6038999999999999e-05 2024-08-06T18:14:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:14:26.836 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019163+0000", 2024-08-06T18:14:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6780999999999996e-05 2024-08-06T18:14:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:14:26.837 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019184+0000", 2024-08-06T18:14:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.1480000000000001e-05 2024-08-06T18:14:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.838 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019354+0000", 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00017021500000000001 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 187.78670020300001, 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000115958, 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:14:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019950+0000", 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.840 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019951+0000", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6499999999999999e-07 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019966+0000", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5464999999999998e-05 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019980+0000", 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3532e-05 2024-08-06T18:14:26.841 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.019987+0000", 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.1060000000000002e-06 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020023+0000", 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6131999999999999e-05 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.842 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020030+0000", 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.3780000000000003e-06 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:11:19.020066+0000", 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.5679999999999997e-05 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:14:26.843 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:14:26.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:26.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:26.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:27.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:27.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:27.175 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:27.175 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:27.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:27.306 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:27.306 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:27.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:27.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:27.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:14:27.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:27 smithi086 ceph-mon[29229]: pgmap v1397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:27.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:27 smithi086 ceph-mon[29229]: osdmap e469: 12 total, 3 up, 11 in 2024-08-06T18:14:27.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:27 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:14:27.601 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:27 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:14:27.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:27.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:14:27.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:27.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:27.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:27.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:27.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:27 smithi117 ceph-mon[32835]: pgmap v1397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:27 smithi117 ceph-mon[32835]: osdmap e469: 12 total, 3 up, 11 in 2024-08-06T18:14:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:27 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:14:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:27 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:14:27.931 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:27.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:27 smithi107 ceph-mon[32381]: pgmap v1397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:27 smithi107 ceph-mon[32381]: osdmap e469: 12 total, 3 up, 11 in 2024-08-06T18:14:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:27 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:14:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:27 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:14:28.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:28.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:28.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:28.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:28.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:28.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:28 smithi117 ceph-mon[32835]: osdmap e470: 12 total, 3 up, 11 in 2024-08-06T18:14:28.635 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:28 smithi107 ceph-mon[32381]: osdmap e470: 12 total, 3 up, 11 in 2024-08-06T18:14:28.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:28.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:28 smithi086 ceph-mon[29229]: osdmap e470: 12 total, 3 up, 11 in 2024-08-06T18:14:28.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:28.912 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:29.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:29.072 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:29.073 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":470,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:27.552246+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:29.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:29.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:29.409 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:29.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:29.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:29.567 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:14:29.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:29.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:29.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:29 smithi086 ceph-mon[29229]: pgmap v1400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:29.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3176140949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:29.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:29.832 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:29.845 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:29.845 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:29.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:29 smithi107 ceph-mon[32381]: pgmap v1400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:29.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:29.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3176140949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:29 smithi117 ceph-mon[32835]: pgmap v1400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3176140949' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:29.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:30.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:30.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:30.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:30.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:30.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:14:30.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:30.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:30.413 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:30.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:30.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:30.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:30 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:30.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:30.711 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:30 smithi086 ceph-mon[29229]: osdmap e471: 12 total, 3 up, 11 in 2024-08-06T18:14:30.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:30 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:30.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:30.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:30 smithi107 ceph-mon[32381]: osdmap e471: 12 total, 3 up, 11 in 2024-08-06T18:14:30.747 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:30.771 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:30 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:30.771 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2719620844' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:30.771 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:30 smithi117 ceph-mon[32835]: osdmap e471: 12 total, 3 up, 11 in 2024-08-06T18:14:30.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:31.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:31.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:31.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:31.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:31.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:31.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:31.698 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:31 smithi117 ceph-mon[32835]: pgmap v1402: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:31.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:31.855 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:31 smithi107 ceph-mon[32381]: pgmap v1402: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:31 smithi086 ceph-mon[29229]: pgmap v1402: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:31.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:31.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:31.982 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:31.983 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:31.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:31.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:31.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:31.999 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:32.000 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:32.001 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:32.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:32.082 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:32.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:32.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:32.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:32.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:32.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/249739810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:32.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:32.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:32.704 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:32.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:32.762 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/249739810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:32.762 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:32.762 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:32.762 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:32.763 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:32.788 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:32.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:14:32.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:32.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/249739810' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:33.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:33.039 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:14:33.117 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:33.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:33.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:33.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:33.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:33.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:33.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:33.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:33.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:14:33.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:33.539 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:33.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:33.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:33.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:33.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:33 smithi107 ceph-mon[32381]: pgmap v1403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:33.781 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:33.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:33.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:33.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:33 smithi086 ceph-mon[29229]: pgmap v1403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:33 smithi117 ceph-mon[32835]: pgmap v1403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:33.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:34.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:34.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:34.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:34.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:34.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:34.395 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:34.395 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:34.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:34.607 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:34.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:34.641 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:34.641 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:34.669 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:34.670 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:34.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:14:34.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:34.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:34.955 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:34.955 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":471,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:29.561042+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:27.294672+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:35.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:35.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:35.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:35.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:35.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:14:35.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:35.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:35.611 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:35.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:35.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:35.831 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:35 smithi107 ceph-mon[32381]: pgmap v1404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:35.831 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:35 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:14:35.831 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:35 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:14:35.831 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/325771248' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:35.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:35 smithi117 ceph-mon[32835]: pgmap v1404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:35.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:35 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:14:35.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:35 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:14:35.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/325771248' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:35.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:35.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:35 smithi086 ceph-mon[29229]: pgmap v1404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:35 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:14:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:35 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:14:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/325771248' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:35.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:35.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:35.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:36.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:36.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:36.245 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:36.245 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:36.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:36.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:36.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:36.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:36 smithi107 ceph-mon[32381]: osdmap e472: 12 total, 3 up, 11 in 2024-08-06T18:14:36.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:36 smithi107 ceph-mon[32381]: osdmap e473: 12 total, 3 up, 11 in 2024-08-06T18:14:36.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:36.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:36 smithi117 ceph-mon[32835]: osdmap e472: 12 total, 3 up, 11 in 2024-08-06T18:14:36.758 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:36 smithi117 ceph-mon[32835]: osdmap e473: 12 total, 3 up, 11 in 2024-08-06T18:14:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:36 smithi086 ceph-mon[29229]: osdmap e472: 12 total, 3 up, 11 in 2024-08-06T18:14:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:36 smithi086 ceph-mon[29229]: osdmap e473: 12 total, 3 up, 11 in 2024-08-06T18:14:36.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:36.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:36.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:36.913 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:37.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:37.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:37.265 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:37.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:37.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:37.561 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:37.561 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:37.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:37.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:37.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:37 smithi117 ceph-mon[32835]: pgmap v1406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:37.813 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:37.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:14:37.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:37.860 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:37 smithi107 ceph-mon[32381]: pgmap v1406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:37.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:37 smithi086 ceph-mon[29229]: pgmap v1406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:37.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:37.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:37.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:38.060 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:38.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:38.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:38.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:38.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:38.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:38.675 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:14:38.675 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:38.676 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:38.676 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:38.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:38.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:38.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:14:38.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:39.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:39.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:39.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:39.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:39.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:39.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:39.610 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:39.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:39.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:39.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:39.749 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: pgmap v1408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:39.750 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:39.750 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:39.750 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: osdmap e474: 12 total, 3 up, 11 in 2024-08-06T18:14:39.750 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:14:39.750 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:39 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:14:39.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: pgmap v1408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:39.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:39.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:39.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: osdmap e474: 12 total, 3 up, 11 in 2024-08-06T18:14:39.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:14:39.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:39 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:14:39.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: pgmap v1408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/169027100' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: osdmap e474: 12 total, 3 up, 11 in 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:14:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:39 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:14:39.957 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:39.957 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:39.957 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:39.969 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:39.970 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:39.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:39.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:39.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:39.987 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:39.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:39.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:40.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:40.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:40.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:14:40.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:40.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:40.608 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:40.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:40.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4190428513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:40.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:40.721 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:40.746 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:40.747 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:40.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:40.853 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4190428513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:40.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:40.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:40.864 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:40.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:40.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4190428513' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:40.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:41.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:41.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:41.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:41.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:41.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:41.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:41.739 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:41.739 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:41 smithi107 ceph-mon[32381]: pgmap v1410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:41.739 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:41 smithi107 ceph-mon[32381]: osdmap e475: 12 total, 3 up, 11 in 2024-08-06T18:14:41.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:14:41.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:41.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:41.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:41 smithi117 ceph-mon[32835]: pgmap v1410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:41.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:41 smithi117 ceph-mon[32835]: osdmap e475: 12 total, 3 up, 11 in 2024-08-06T18:14:41.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:41.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:41 smithi086 ceph-mon[29229]: pgmap v1410: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:41 smithi086 ceph-mon[29229]: osdmap e475: 12 total, 3 up, 11 in 2024-08-06T18:14:41.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:42.104 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:42.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:42.200 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:42.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:42.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:42.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:42.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:42.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:42.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:42.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:14:42.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:42.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:42.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:14:42.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:43.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:43.145 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:43.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:43.207 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:43.207 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:43.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:43.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:43.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:43.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:43.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:43.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:43.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:43.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:43 smithi117 ceph-mon[32835]: pgmap v1412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:43.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:43 smithi117 ceph-mon[32835]: osdmap e476: 12 total, 3 up, 11 in 2024-08-06T18:14:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:43 smithi086 ceph-mon[29229]: pgmap v1412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:43 smithi086 ceph-mon[29229]: osdmap e476: 12 total, 3 up, 11 in 2024-08-06T18:14:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:43 smithi107 ceph-mon[32381]: pgmap v1412: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:43 smithi107 ceph-mon[32381]: osdmap e476: 12 total, 3 up, 11 in 2024-08-06T18:14:43.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:44.030 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:44.030 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":476,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:42.602556+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:44.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:44.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:44.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:44.475 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:44.475 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:44.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:44.583 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:44.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:44.672 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1795083771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:44.768 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1795083771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:44.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:44.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:44.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:44.867 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1795083771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:44.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:45.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:45.061 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:45.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:45.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:14:45.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:45.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:45.484 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:45.587 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:45.617 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:14:45.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:45 smithi086 ceph-mon[29229]: pgmap v1414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:45.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:45 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:14:45.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:45 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:14:45.710 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:45.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:45.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:45 smithi117 ceph-mon[32835]: pgmap v1414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:45.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:45 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:14:45.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:45 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:14:45.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:45.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:45 smithi107 ceph-mon[32381]: pgmap v1414: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:45 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:14:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:45 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:14:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:14:45.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:46.017 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:46.018 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:46.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:46.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:46.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:46.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:46.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:46.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:46.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:46.659 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:46 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:46.659 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:46.659 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:46 smithi086 ceph-mon[29229]: osdmap e477: 12 total, 3 up, 11 in 2024-08-06T18:14:46.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:46.792 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:46 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:46.792 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:46.792 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:46 smithi117 ceph-mon[32835]: osdmap e477: 12 total, 3 up, 11 in 2024-08-06T18:14:46.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:46.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:46 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:14:46.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1688798264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:14:46.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:46 smithi107 ceph-mon[32381]: osdmap e477: 12 total, 3 up, 11 in 2024-08-06T18:14:47.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:47.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:47.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:47.094 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:47.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:47.198 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:47.198 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:47.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:47.343 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:47.343 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:47.440 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:47.440 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:47.441 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:47.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:47.456 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:47.457 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:47.457 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:47.457 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:47.458 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:47.475 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:47.477 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:47.477 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:47.477 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:47.477 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:47.478 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:47.478 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:47.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:47.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:47.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:47.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:47.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:14:47.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:47.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:47.859 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:47 smithi117 ceph-mon[32835]: pgmap v1416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:47.859 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2227385009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:47 smithi107 ceph-mon[32381]: pgmap v1416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2227385009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:47.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:48.029 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:48.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:48.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:47 smithi086 ceph-mon[29229]: pgmap v1416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2227385009' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:48.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:48.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:48.200 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:48.200 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:48.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:48.277 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:48.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:48.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:14:48.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:48.519 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:48.519 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:48.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:48.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:48.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.876 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.876 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:48.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:49.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:49.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:49.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:49.435 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:49.448 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:49.449 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:49.523 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:49.547 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:49.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:49.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:49.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:49.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:49 smithi107 ceph-mon[32381]: pgmap v1417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:49.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:49.880 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:49 smithi086 ceph-mon[29229]: pgmap v1417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:49.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:50.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:49 smithi117 ceph-mon[32835]: pgmap v1417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:50.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:14:50.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:50.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:50.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:50.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:50.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:14:50.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:50.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:50.532 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:50.589 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:50.589 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:50.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:50.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:50.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:50.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:50.858 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:50.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:50.995 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:50.995 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":477,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:45.614244+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:51.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:51.056 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:51.056 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:51.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:51.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:51.430 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:51.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:51.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:51.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:51.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:51.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:51.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:51.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:51.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:51 smithi086 ceph-mon[29229]: pgmap v1418: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:51.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3185059711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:51.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:51 smithi117 ceph-mon[32835]: pgmap v1418: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:51.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3185059711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:51.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:51 smithi107 ceph-mon[32381]: pgmap v1418: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3185059711' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:52.065 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:52.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:52.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:52.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:52.487 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:52.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:14:52.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:52.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:14:52.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:52.778 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:52.779 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:52.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:53.000 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:53.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:53.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:53.180 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:53.180 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:53.180 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:53.181 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:53.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:53.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:53.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:53.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:53.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:53.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:53.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:53 smithi107 ceph-mon[32381]: pgmap v1419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:53.764 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:53 smithi117 ceph-mon[32835]: pgmap v1419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:53.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:53 smithi086 ceph-mon[29229]: pgmap v1419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:53.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:53.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:54.040 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:14:54.040 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:14:54.041 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:14:54.057 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:14:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:14:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:14:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:14:54.058 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:14:54.075 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:14:54.075 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:14:54.076 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:54.076 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:54.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:14:54.076 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:14:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:14:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:14:54.077 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:14:54.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:54.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:54.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:54.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:54.500 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:54.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:54.635 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:54.636 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:54.681 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:14:54.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:54.829 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:54.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:54.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:54.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/160459823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:54.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:54.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:54.859 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:14:54.860 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:14:54.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:54.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:54.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/160459823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:54.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:54.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:14:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/160459823' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:14:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:55.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:55.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:55.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:55.309 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:55.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:14:55.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:55.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:55.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:55.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:55.604 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:55.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:55.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:55 smithi086 ceph-mon[29229]: pgmap v1420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:55.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:55.713 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:55 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:55.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:55.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:55 smithi086 ceph-mon[29229]: osdmap e478: 12 total, 3 up, 11 in 2024-08-06T18:14:55.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:55.932 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:14:55.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:55 smithi117 ceph-mon[32835]: pgmap v1420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:55.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:55.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:55 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:55.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:55.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:55 smithi117 ceph-mon[32835]: osdmap e478: 12 total, 3 up, 11 in 2024-08-06T18:14:55.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:55 smithi107 ceph-mon[32381]: pgmap v1420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:14:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:55 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:14:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1377168548' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:14:55.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:55 smithi107 ceph-mon[32381]: osdmap e478: 12 total, 3 up, 11 in 2024-08-06T18:14:55.998 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:55.998 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:56.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:56.191 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:56.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:56.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:56.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:56.533 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:56.533 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:56.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:56.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:56.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:14:56.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:56.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:56.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:57.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:14:57.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:57.218 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:57.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:57.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:14:57.328 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:14:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:14:57.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:57.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:57.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: pgmap v1422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: osdmap e479: 12 total, 3 up, 11 in 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.606 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:57.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:57.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: pgmap v1422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: osdmap e479: 12 total, 3 up, 11 in 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:14:57.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:14:57.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: pgmap v1422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: osdmap e479: 12 total, 3 up, 11 in 2024-08-06T18:14:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:14:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:58.112 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:58.112 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:14:58.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:58.201 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:14:58.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:14:58.275 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:14:58.275 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":479,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:14:56.552011+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:13:50.250091+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:14:58.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:14:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:14:58.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:14:58.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:14:58.556 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:14:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:14:58.621 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:14:58.621 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:14:58.621 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:58 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:58.621 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/105514191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:58.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:58.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:58.789 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:58 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:14:58.789 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:58 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:14:58.789 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:58 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:58.789 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/105514191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:58.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:14:58.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:14:58.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:58 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:14:58.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/105514191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:14:58.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:14:59.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:59.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:14:59.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:14:59.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:14:59.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:14:59.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:59.761 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:14:59.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:14:59.847 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:14:59 smithi117 ceph-mon[32835]: pgmap v1424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:14:59 smithi086 ceph-mon[29229]: pgmap v1424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:59.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:14:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:14:59 smithi107 ceph-mon[32381]: pgmap v1424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:14:59.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:14:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:14:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:00.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:00.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:00.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:00.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:00.485 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:00.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:15:00.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:00.713 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:00.713 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:00.733 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:15:00.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:00.778 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:00.778 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:00.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:01.023 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:01.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:01.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:01.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:01.314 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:01.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:01.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:01.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:01 smithi086 ceph-mon[29229]: pgmap v1425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:01.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:01.709 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:01 smithi117 ceph-mon[32835]: pgmap v1425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:01.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:01.827 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:01.827 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:01.843 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:01 smithi107 ceph-mon[32381]: pgmap v1425: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:01.844 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:01.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:01.861 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:01.862 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:01.863 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:01.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:02.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:02.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:02.232 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:02.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:02.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:02.502 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:02.502 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:02.502 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:02.502 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:02.533 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:02.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:02.615 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:02.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:02.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/420909297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:02.754 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/420909297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:02.906 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:02.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:15:02.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/420909297' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:02.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:02.911 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:02.911 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:02.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:03.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:03.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:03.157 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:15:03.173 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:03.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:03.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:03.442 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:03.442 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:03.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:03.568 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:03.568 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:03.572 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:15:03.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:03.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:03.858 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:03.858 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:03.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:03.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: pgmap v1426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:03.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:15:03.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:15:03.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: osdmap e480: 12 total, 3 up, 11 in 2024-08-06T18:15:03.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: pgmap v1426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: osdmap e480: 12 total, 3 up, 11 in 2024-08-06T18:15:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: pgmap v1426: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:15:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:15:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: osdmap e480: 12 total, 3 up, 11 in 2024-08-06T18:15:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:03.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:04.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:04.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:04.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:04.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:04.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:04.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:04.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:04.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:04.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:15:04.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:04 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:04.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:04 smithi107 ceph-mon[32381]: osdmap e481: 12 total, 3 up, 11 in 2024-08-06T18:15:04.730 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:04.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:04 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:04.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:04 smithi086 ceph-mon[29229]: osdmap e481: 12 total, 3 up, 11 in 2024-08-06T18:15:04.785 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:04.785 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:04 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/336228775' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:04 smithi117 ceph-mon[32835]: osdmap e481: 12 total, 3 up, 11 in 2024-08-06T18:15:04.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:05.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:05.078 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:05.078 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:05.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:05.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:05.359 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:05.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:15:05.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:05.465 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:05.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:05.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:05 smithi117 ceph-mon[32835]: pgmap v1429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:05.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:05.677 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:15:05.678 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:05 smithi086 ceph-mon[29229]: pgmap v1429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:05.678 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:05.678 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:05.679 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":481,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:03.569340+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:15:05.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:05.835 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:05 smithi107 ceph-mon[32381]: pgmap v1429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:05.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:06.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:06.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:06.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:06.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:06.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:06.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:06.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:06.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1335037414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:06.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1335037414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:06.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:06.904 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1335037414' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:06.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:15:07.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:07.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:07.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:07.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:07.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:07.302 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:07.302 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:07.387 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:07.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:07.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:07.584 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:07.584 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:07.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:07.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:07 smithi107 ceph-mon[32381]: pgmap v1430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:07.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:07.843 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:07 smithi117 ceph-mon[32835]: pgmap v1430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:07.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:07 smithi086 ceph-mon[29229]: pgmap v1430: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:07.855 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:07.953 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:08.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:08.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:08.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:08.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:08.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:08.639 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:08.639 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:08.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:08.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:08.906 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:08.906 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:09.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:09.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:09.297 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:09.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:09.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:09.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:09.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:09.760 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:09.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:09.863 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:09 smithi107 ceph-mon[32381]: pgmap v1431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:09.863 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:09.863 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:09.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:09 smithi086 ceph-mon[29229]: pgmap v1431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:09.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:09 smithi117 ceph-mon[32835]: pgmap v1431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:09.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:10.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:10.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:10.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:10.394 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:10.394 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:10.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:10.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:10.576 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:10.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:15:10.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:10.661 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:10.661 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:10.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:10.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:10.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:10.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:10.978 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:10.978 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:11.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:11.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:11.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:11.310 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:11.310 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:11.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:11.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:11.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:11.786 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:11.786 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:11.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:11.804 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:11.806 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:11.806 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:11.807 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:11.809 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:11.834 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:11.835 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:11.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:11.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:11.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:11.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:11.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:11.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:11.863 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:11 smithi117 ceph-mon[32835]: pgmap v1432: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:11.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:11 smithi086 ceph-mon[29229]: pgmap v1432: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:11 smithi107 ceph-mon[32381]: pgmap v1432: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:11.972 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:12.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:12.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:12.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:12.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:12.599 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:15:12.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:12.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:12.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:12.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:12.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:12.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1419172245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:12.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:12.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:12.850 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:12.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1419172245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:12.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1419172245' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:12.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:13.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:13.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:13.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:13.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:13.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:13.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:13.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:13.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:13 smithi086 ceph-mon[29229]: pgmap v1433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:13.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:13 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:13.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:13.777 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:13 smithi086 ceph-mon[29229]: osdmap e482: 12 total, 3 up, 11 in 2024-08-06T18:15:13.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:13.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:13.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:13 smithi117 ceph-mon[32835]: pgmap v1433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:13 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:13 smithi117 ceph-mon[32835]: osdmap e482: 12 total, 3 up, 11 in 2024-08-06T18:15:13.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:13.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:13 smithi107 ceph-mon[32381]: pgmap v1433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:13.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:13 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:13.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1969258951' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:13.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:13 smithi107 ceph-mon[32381]: osdmap e482: 12 total, 3 up, 11 in 2024-08-06T18:15:13.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:14.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:14.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:14.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:14.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:14.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:14.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:14.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:15:14.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:14.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:14.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:14.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:14.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:15.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:15.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:15.206 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:15.207 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:15.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:15.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:15.597 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:15.597 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:15.597 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:15.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:15:15.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:15.755 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:15 smithi117 ceph-mon[32835]: pgmap v1435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:15.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:15 smithi086 ceph-mon[29229]: pgmap v1435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:15 smithi107 ceph-mon[32381]: pgmap v1435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:16.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:16.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:16.140 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:16.140 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:16.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:16.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:16.484 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:16.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:16.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:16.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:16.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:17.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:17.190 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:17.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:15:17.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:17.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:17.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:17.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:17.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:17.455 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:17.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:17.485 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:17.487 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:17.558 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:17.559 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:17.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:17.615 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:15:17.615 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":483,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:16.557424+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:15:17.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:17.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:17 smithi086 ceph-mon[29229]: pgmap v1436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:17.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:17 smithi086 ceph-mon[29229]: osdmap e483: 12 total, 3 up, 11 in 2024-08-06T18:15:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:17 smithi117 ceph-mon[32835]: pgmap v1436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:17 smithi117 ceph-mon[32835]: osdmap e483: 12 total, 3 up, 11 in 2024-08-06T18:15:17.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:17 smithi107 ceph-mon[32381]: pgmap v1436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:17.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:17 smithi107 ceph-mon[32381]: osdmap e483: 12 total, 3 up, 11 in 2024-08-06T18:15:18.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:18.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:18.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:18.417 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:18.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:18.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:18.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:18.846 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2858266679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:18.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:18.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:18.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2858266679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:18.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2858266679' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:18.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:19.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:19.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:19.317 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:19.317 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:19.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:19.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:19.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:19 smithi086 ceph-mon[29229]: pgmap v1438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:19.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:19.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:19 smithi117 ceph-mon[32835]: pgmap v1438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:19.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:19.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:19 smithi107 ceph-mon[32381]: pgmap v1438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:19.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:19.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:20.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:20.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:20.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:20.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:20.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:20.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:20.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:20.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:21.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:21.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:21.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:21.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:21.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:21.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:21.581 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:21.581 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:21.598 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:21.599 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:21.599 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:21.599 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:21.599 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:21.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:21.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:21.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:21.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:21.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:21 smithi086 ceph-mon[29229]: pgmap v1439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:21.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1459121052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:21.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:21 smithi107 ceph-mon[32381]: pgmap v1439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:21.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1459121052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:21 smithi117 ceph-mon[32835]: pgmap v1439: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1459121052' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:21.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:22.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:22.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:22.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:22.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:22.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:22.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:22.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:22.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:22.565 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:22.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:22.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:22.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:23.001 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:23.001 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:23.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:23.207 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:23.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:23.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:23.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:23.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:23.461 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:23.462 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:23.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:23.574 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:15:23.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:23.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:23 smithi107 ceph-mon[32381]: pgmap v1440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:23.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:23.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:23.769 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:23 smithi086 ceph-mon[29229]: pgmap v1440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:23.769 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:23.770 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:23.770 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:23.796 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:23 smithi117 ceph-mon[32835]: pgmap v1440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:23.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:24.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:24.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:24.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:24.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:24.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:24.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:24.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:15:24.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:24.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:24.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:24.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:24 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:15:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:24.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:24 smithi107 ceph-mon[32381]: osdmap e484: 12 total, 3 up, 11 in 2024-08-06T18:15:24.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:24.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:24 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:15:24.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:24.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:24 smithi086 ceph-mon[29229]: osdmap e484: 12 total, 3 up, 11 in 2024-08-06T18:15:24.860 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:24 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:15:24.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2864233211' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:24.861 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:24 smithi117 ceph-mon[32835]: osdmap e484: 12 total, 3 up, 11 in 2024-08-06T18:15:24.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:25.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:25.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:25.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:25.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:25.349 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:25.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:15:25.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:25.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:25.585 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:25.585 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:25.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:25.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:25.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:25 smithi086 ceph-mon[29229]: pgmap v1442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:25.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:25 smithi117 ceph-mon[32835]: pgmap v1442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:25.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:25 smithi107 ceph-mon[32381]: pgmap v1442: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:25.963 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:26.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:26.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:26.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:26.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:26.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:26.602 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:26.602 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:26.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:15:26.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:26.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:26.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:27.105 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:15:27.105 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":485,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:26.560086+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:26.578489+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:15:27.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:27.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:27.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:27.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:27.640 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:27.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:27.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:27.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: osdmap e485: 12 total, 3 up, 11 in 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:15:27.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2898468407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: osdmap e485: 12 total, 3 up, 11 in 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:15:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:15:27.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:15:27.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:15:27.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2898468407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:27.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: osdmap e485: 12 total, 3 up, 11 in 2024-08-06T18:15:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:15:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2898468407' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:28.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:28.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:28.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:28.424 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:28.428 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:28.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:28.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:28.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:28.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:28.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:29.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:29.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:29.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:29.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:29.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:29.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:29.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:29 smithi086 ceph-mon[29229]: pgmap v1445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:29 smithi117 ceph-mon[32835]: pgmap v1445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:29.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:29.951 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:29 smithi107 ceph-mon[32381]: pgmap v1445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:30.061 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:30.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:30.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:30.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:30.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:30.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:30.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:30.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:30.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:30.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:30.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:31.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:31.138 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:31.138 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:31.153 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:31.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:31.172 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:31.172 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:31.173 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:31.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:31.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:31.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:31.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:31.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:31.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:31 smithi107 ceph-mon[32381]: pgmap v1446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:31.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1388412122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:31.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.819 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:31 smithi086 ceph-mon[29229]: pgmap v1446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1388412122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:31 smithi117 ceph-mon[32835]: pgmap v1446: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1388412122' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:31.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:32.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:32.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:32.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:32.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:32.322 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:32.323 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:32.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:32.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:32.571 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:15:32.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:32.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:32.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:32.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:32.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:15:32.834 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:32.834 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:32.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:15:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:15:32.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:33.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:33.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:33.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:33.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:33.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:33.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:33.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:33 smithi107 ceph-mon[32381]: pgmap v1447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:33 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:15:33.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:33 smithi107 ceph-mon[32381]: osdmap e486: 12 total, 3 up, 11 in 2024-08-06T18:15:33.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:33.781 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:33.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:33 smithi086 ceph-mon[29229]: pgmap v1447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:33.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:33 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:33.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:15:33.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:33 smithi086 ceph-mon[29229]: osdmap e486: 12 total, 3 up, 11 in 2024-08-06T18:15:33.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:33.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:33 smithi117 ceph-mon[32835]: pgmap v1447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:33.823 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:33 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:33.824 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/486121402' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:15:33.824 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:33 smithi117 ceph-mon[32835]: osdmap e486: 12 total, 3 up, 11 in 2024-08-06T18:15:33.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:34.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:34.066 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:15:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:15:34.066 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:15:34.066 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:15:34.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:34.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:15:34.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:34.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:34.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:34.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:34.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:34.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:15:34.748 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:34.762 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:34.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:34.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:35.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:35.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:35.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:35.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:35.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:15:35.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:35.583 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:35 smithi086 ceph-mon[29229]: pgmap v1449: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:35.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:35.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:35.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:35.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:35 smithi117 ceph-mon[32835]: pgmap v1449: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:35 smithi107 ceph-mon[32381]: pgmap v1449: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:35.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:36.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:36.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:36.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:36.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:36.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:36.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:36.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:36.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:36.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:36.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:36.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:36.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:36.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:37.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:37.062 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:37.063 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:37.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:37.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:37.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:37.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:37.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:37 smithi107 ceph-mon[32381]: pgmap v1450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:37.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:37 smithi107 ceph-mon[32381]: osdmap e487: 12 total, 3 up, 11 in 2024-08-06T18:15:37.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:37 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:15:37.599 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:37 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:15:37.694 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:37.694 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:37.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:37 smithi086 ceph-mon[29229]: pgmap v1450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:37.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:37 smithi086 ceph-mon[29229]: osdmap e487: 12 total, 3 up, 11 in 2024-08-06T18:15:37.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:37 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:15:37.694 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:37 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:15:37.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:37.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:37.869 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:37 smithi117 ceph-mon[32835]: pgmap v1450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:37.869 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:37 smithi117 ceph-mon[32835]: osdmap e487: 12 total, 3 up, 11 in 2024-08-06T18:15:37.869 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:37 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:15:37.869 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:37 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:15:37.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:38.013 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:38.014 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:38.014 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:38.030 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:38.031 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:38.031 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:38.032 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:38.032 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:38.048 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:38.049 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:38.049 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:38.049 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:38.049 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:38.050 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:38.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:38.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:38.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:38.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:38.503 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:38.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:38.619 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:38.620 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:38.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:38.666 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:15:38.666 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":488,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:37.615707+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:14:34.788338+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:15:38.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:38 smithi117 ceph-mon[32835]: osdmap e488: 12 total, 3 up, 11 in 2024-08-06T18:15:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2047057723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:38.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:38 smithi086 ceph-mon[29229]: osdmap e488: 12 total, 3 up, 11 in 2024-08-06T18:15:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2047057723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:38.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:38.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:38.951 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:38 smithi107 ceph-mon[32381]: osdmap e488: 12 total, 3 up, 11 in 2024-08-06T18:15:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2047057723' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:39.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:39.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:39.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:39.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:39.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:39.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:39.745 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:39.745 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:39.745 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:39.745 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:39 smithi107 ceph-mon[32381]: pgmap v1453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:39.745 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3758493861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:39.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:39.862 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:39.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:39.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:39 smithi086 ceph-mon[29229]: pgmap v1453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3758493861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:39 smithi117 ceph-mon[32835]: pgmap v1453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3758493861' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:39.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:40.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:40.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:40.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:40.263 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:40.264 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:40.264 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:40.264 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:40.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:40.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:40.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:40.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:40.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:40.822 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:40.822 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:40.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:40.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:15:41.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:41.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:41.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:41.350 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:41.351 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:41.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:41.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:41.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:41.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:41.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:41.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:41.902 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:41 smithi117 ceph-mon[32835]: pgmap v1454: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:41.902 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:41 smithi117 ceph-mon[32835]: osdmap e489: 12 total, 3 up, 11 in 2024-08-06T18:15:41.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:41 smithi086 ceph-mon[29229]: pgmap v1454: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:41.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:41 smithi086 ceph-mon[29229]: osdmap e489: 12 total, 3 up, 11 in 2024-08-06T18:15:41.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:41.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:41 smithi107 ceph-mon[32381]: pgmap v1454: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:41 smithi107 ceph-mon[32381]: osdmap e489: 12 total, 3 up, 11 in 2024-08-06T18:15:42.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:15:42.071 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:15:42.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:42.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:42.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:42.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:42.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:42.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:42.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:43.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:43.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:43.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:43.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:43.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:43.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:15:43.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:43.512 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:43.512 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:43.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:43.651 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:43 smithi107 ceph-mon[32381]: pgmap v1456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:43.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:43.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:43 smithi086 ceph-mon[29229]: pgmap v1456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:43 smithi117 ceph-mon[32835]: pgmap v1456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:43.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:43.921 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:43.921 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:43.923 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:43.965 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:43.966 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:44.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:44.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:44.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:44.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:44.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:44.631 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:15:44.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:44.718 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:44.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:15:44.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:45.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:45.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:45.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:45.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:45.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:45.746 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:45.747 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:45.747 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:45.747 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:45.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: pgmap v1457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:45.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:15:45.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:15:45.747 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:45.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:45.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:45 smithi086 ceph-mon[29229]: osdmap e490: 12 total, 3 up, 11 in 2024-08-06T18:15:45.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: pgmap v1457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:15:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:15:45.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:45.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:45.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:45 smithi117 ceph-mon[32835]: osdmap e490: 12 total, 3 up, 11 in 2024-08-06T18:15:45.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: pgmap v1457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:15:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:15:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3270953500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:15:45.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:45 smithi107 ceph-mon[32381]: osdmap e490: 12 total, 3 up, 11 in 2024-08-06T18:15:46.046 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:46.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:46.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:46.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:46.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:46.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:46.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:46.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:47.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:47.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:47.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:47.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:47.566 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:47.566 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:47.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:47.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:47.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:47 smithi086 ceph-mon[29229]: pgmap v1459: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:47.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:47 smithi086 ceph-mon[29229]: osdmap e491: 12 total, 3 up, 11 in 2024-08-06T18:15:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:47 smithi117 ceph-mon[32835]: pgmap v1459: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:47 smithi117 ceph-mon[32835]: osdmap e491: 12 total, 3 up, 11 in 2024-08-06T18:15:47.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:47 smithi107 ceph-mon[32381]: pgmap v1459: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:47 smithi107 ceph-mon[32381]: osdmap e491: 12 total, 3 up, 11 in 2024-08-06T18:15:48.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:48.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:15:48.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:48.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:48.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:48.620 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:48.620 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:48.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:48.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:48.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:48.856 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:48.871 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:48.873 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:48.873 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:48.876 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:48.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:48.877 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:48.878 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:48.889 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:48.890 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:48.890 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:48.890 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:48.890 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:48.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:48.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:48.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:49.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:49.082 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:15:49.082 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:387,from:6,reservation_nonce:15)", 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 422.52990704600001, 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000152874, 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.083 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:15:49.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552879+0000", 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.579e-06 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552889+0000", 2024-08-06T18:15:49.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6840000000000008e-06 2024-08-06T18:15:49.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552910+0000", 2024-08-06T18:15:49.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1277e-05 2024-08-06T18:15:49.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552922+0000", 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1421e-05 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552990+0000", 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8114999999999994e-05 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553003+0000", 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3453e-05 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553031+0000", 2024-08-06T18:15:49.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7345e-05 2024-08-06T18:15:49.090 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.090 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:387/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 422.52911475100001, 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125494, 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553671+0000", 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4799999999999999e-07 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553674+0000", 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8e-06 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553701+0000", 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6699999999999998e-05 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553708+0000", 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0559999999999999e-06 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553764+0000", 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5810000000000003e-05 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553772+0000", 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5560000000000002e-06 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553796+0000", 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4124000000000001e-05 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:387,from:6,reservation_nonce:0)", 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 422.52832979200002, 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011286299999999999, 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.096 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554456+0000", 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6100000000000002e-07 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554458+0000", 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.582e-06 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554463+0000", 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0520000000000001e-06 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554471+0000", 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7999999999999999e-06 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554549+0000", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8207000000000005e-05 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554559+0000", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.023e-05 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554568+0000", 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6309999999999992e-06 2024-08-06T18:15:49.100 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:423,from:6,reservation_nonce:17)", 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 270.06390798699999, 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014934700000000001, 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.101 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018879+0000", 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1619999999999998e-06 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018888+0000", 2024-08-06T18:15:49.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1570000000000006e-06 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018906+0000", 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7773000000000001e-05 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018917+0000", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1262e-05 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018984+0000", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6549000000000003e-05 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018999+0000", 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5593000000000001e-05 2024-08-06T18:15:49.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019026+0000", 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6851000000000001e-05 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:423/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:15:49.106 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 270.06307881399999, 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014629899999999999, 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019707+0000", 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6499999999999999e-07 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019740+0000", 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3853999999999998e-05 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019750+0000", 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019761+0000", 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1636e-05 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019809+0000", 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019820+0000", 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1606999999999999e-05 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019852+0000", 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2240000000000003e-05 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:423,from:6,reservation_nonce:0)", 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 270.06212301900001, 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000163024, 2024-08-06T18:15:49.111 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.86e-07 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020664+0000", 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.579e-06 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020685+0000", 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1350999999999999e-05 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020689+0000", 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2910000000000001e-06 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020805+0000", 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011613 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020816+0000", 2024-08-06T18:15:49.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.048e-05 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020825+0000", 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0070000000000003e-06 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:441,from:2,reservation_nonce:16)", 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 212.462868109, 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011107860000000001, 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:15:49.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619918+0000", 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619927+0000", 2024-08-06T18:15:49.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:15:49.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619942+0000", 2024-08-06T18:15:49.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5214999999999999e-05 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619950+0000", 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9519999999999994e-06 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620000+0000", 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9979999999999999e-05 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:15:49.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620004+0000", 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0589999999999996e-06 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620978+0000", 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097387000000000005 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620993+0000", 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4676e-05 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621028+0000", 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4950999999999998e-05 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.125 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:441/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 212.46126076900001, 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5687999999999997e-05, 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:15:49.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621525+0000", 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8700000000000005e-07 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621530+0000", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3519999999999997e-06 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621541+0000", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0828e-05 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621546+0000", 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.383e-06 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621585+0000", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8364000000000001e-05 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621591+0000", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9649999999999996e-06 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621610+0000", 2024-08-06T18:15:49.129 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9108999999999999e-05 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:441,from:2,reservation_nonce:0)", 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 212.46062449300001, 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111129, 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.130 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:15:49.131 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622161+0000", 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7599999999999998e-07 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622163+0000", 2024-08-06T18:15:49.132 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.951e-06 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622167+0000", 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8650000000000003e-06 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622170+0000", 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5260000000000002e-06 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.133 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622240+0000", 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9622000000000005e-05 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622257+0000", 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7754e-05 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.134 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622272+0000", 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4134999999999999e-05 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:457,from:2,reservation_nonce:17)", 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.72994854500001, 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010675699999999999, 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.135 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.136 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352837+0000", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7300000000000005e-07 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352842+0000", 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2769999999999999e-06 2024-08-06T18:15:49.137 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352852+0000", 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352859+0000", 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6959999999999999e-06 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.138 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352906+0000", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7364999999999998e-05 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352919+0000", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3159999999999999e-05 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352943+0000", 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3828e-05 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.139 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:457/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.72930874900001, 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3411000000000004e-05, 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.140 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353477+0000", 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4199999999999995e-07 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353481+0000", 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5820000000000003e-06 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353493+0000", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1887e-05 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353499+0000", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6010000000000002e-06 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353543+0000", 2024-08-06T18:15:49.143 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3927000000000001e-05 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353549+0000", 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5549999999999999e-06 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.144 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.145 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353570+0000", 2024-08-06T18:15:49.145 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0217e-05 2024-08-06T18:15:49.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.145 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.145 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:457,from:2,reservation_nonce:0)", 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 130.728588414, 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012299899999999999, 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.146 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.147 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.39e-07 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354199+0000", 2024-08-06T18:15:49.148 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5150000000000001e-06 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354206+0000", 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0090000000000001e-06 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354219+0000", 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2714999999999999e-05 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.149 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354300+0000", 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1031000000000004e-05 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354310+0000", 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0359e-05 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.150 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354320+0000", 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0310000000000004e-06 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:476,from:2,reservation_nonce:18)", 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.661727583000001, 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012406, 2024-08-06T18:15:49.151 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.152 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421059+0000", 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.471e-06 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421067+0000", 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0600000000000008e-06 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.153 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421082+0000", 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305e-05 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421093+0000", 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.154 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421142+0000", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8767999999999998e-05 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421156+0000", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4027e-05 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421181+0000", 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5871e-05 2024-08-06T18:15:49.155 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:476/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.661126431, 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119988, 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.156 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.157 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4099999999999999e-07 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.158 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421663+0000", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3649999999999997e-06 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421673+0000", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0149000000000001e-05 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421681+0000", 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9920000000000007e-06 2024-08-06T18:15:49.159 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421726+0000", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4246999999999998e-05 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421737+0000", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1796e-05 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421778+0000", 2024-08-06T18:15:49.160 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0998000000000003e-05 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:476,from:2,reservation_nonce:0)", 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 64.660463012999998, 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111465, 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:15:49.161 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.162 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2300000000000002e-07 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422325+0000", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422331+0000", 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2530000000000001e-06 2024-08-06T18:15:49.163 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422336+0000", 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.558e-06 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422406+0000", 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9059000000000004e-05 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:15:49.164 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422420+0000", 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422433+0000", 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3499000000000001e-05 2024-08-06T18:15:49.165 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.166 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.166 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.166 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:15:49.166 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:15:49.166 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:15:49.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:49.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:49.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:49.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:49.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:49.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:49.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:49.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:49.856 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:49.856 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:49.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:49 smithi086 ceph-mon[29229]: pgmap v1461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:49.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1901522665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:49.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.867 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:49 smithi107 ceph-mon[32381]: pgmap v1461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:49.867 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1901522665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:49.867 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.867 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:49 smithi117 ceph-mon[32835]: pgmap v1461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1901522665' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:49.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:50.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:50.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:50.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:50.230 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:15:50.231 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:15:50.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:50.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:50.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:50.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:50.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:50.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:50.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:15:50.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:50.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:50.805 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:50.805 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:50.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:51.015 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:51.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:51.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:51.187 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:15:51.187 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":491,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:46.565280+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:15:51.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:51.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:51.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:51.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:51.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:51.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:51.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:51 smithi117 ceph-mon[32835]: pgmap v1462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:51.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3402697024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:51.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:51.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:51 smithi086 ceph-mon[29229]: pgmap v1462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:51.734 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3402697024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:51.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:51.734 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:51.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:51.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:51.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:51 smithi107 ceph-mon[32381]: pgmap v1462: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3402697024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:15:51.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:52.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:52.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:52.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:52.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:15:52.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:52.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:52.395 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:52.396 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:52.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:52.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:52.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:52.513 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:52.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:52.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:52.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:52.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:53.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:53.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:53.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:53.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:53.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:53.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:53 smithi107 ceph-mon[32381]: pgmap v1463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:53.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:53.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:53.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:53.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:53 smithi086 ceph-mon[29229]: pgmap v1463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:53 smithi117 ceph-mon[32835]: pgmap v1463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:53.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:54.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:54.193 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:54.193 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:54.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:54.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:54.467 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:54.467 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:54.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:54.563 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:54.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:54.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:54.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:54.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:15:55.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:55.112 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:15:55.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:55.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:55.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:55.587 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:15:55.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:55 smithi107 ceph-mon[32381]: pgmap v1464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:55.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:55.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:55.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:55 smithi086 ceph-mon[29229]: pgmap v1464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:55.829 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:55.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:55.830 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:55.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:55 smithi117 ceph-mon[32835]: pgmap v1464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:15:55.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:56.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:56.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:56.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:56.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:56.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:56.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:56.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:56.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:56.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:56.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:56.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:56.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:56 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:56 smithi086 ceph-mon[29229]: osdmap e492: 12 total, 3 up, 11 in 2024-08-06T18:15:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:56 smithi086 ceph-mon[29229]: osdmap e493: 12 total, 3 up, 11 in 2024-08-06T18:15:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:56 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:56 smithi117 ceph-mon[32835]: osdmap e492: 12 total, 3 up, 11 in 2024-08-06T18:15:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:56 smithi117 ceph-mon[32835]: osdmap e493: 12 total, 3 up, 11 in 2024-08-06T18:15:56.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:56 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:15:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1394110841' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:15:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:56 smithi107 ceph-mon[32381]: osdmap e492: 12 total, 3 up, 11 in 2024-08-06T18:15:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:56 smithi107 ceph-mon[32381]: osdmap e493: 12 total, 3 up, 11 in 2024-08-06T18:15:57.001 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:15:57.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:15:57.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:57.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:57.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:15:57.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:57.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:57.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:57.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:57.610 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:57 smithi086 ceph-mon[29229]: pgmap v1466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:57.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:57.735 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:57.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:15:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:57 smithi117 ceph-mon[32835]: pgmap v1466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:57.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:57 smithi107 ceph-mon[32381]: pgmap v1466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 324 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:57.888 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:57.888 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:57.954 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:15:57.955 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:15:57.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:15:57.962 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:15:57.962 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:15:57.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:15:57.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:15:57.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:15:57.979 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:15:57.979 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:15:57.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:15:57.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:15:57.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:15:57.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:15:57.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:15:57.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:15:58.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:58.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:15:58.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:15:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:15:58.222 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:15:58.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:15:58.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:58.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:15:58.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:15:58.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:15:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:15:58.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:15:58.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:58.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:58.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3839773041' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:58.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:58.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3839773041' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:58.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3839773041' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:15:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:58.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:59.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:15:59.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:15:59.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:15:59.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:15:59.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:15:59.584 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:15:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:15:59.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:15:59.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:15:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:15:59 smithi086 ceph-mon[29229]: pgmap v1468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:59.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:15:59.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:59.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:15:59 smithi117 ceph-mon[32835]: pgmap v1468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:15:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:15:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:15:59 smithi107 ceph-mon[32381]: pgmap v1468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:00.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:00.025 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:00.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:00.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:00.417 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:00.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:00.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:00.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:00.567 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:00.568 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:00.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:00.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:00.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:00 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:16:00.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:00 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:16:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:00 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:16:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:00 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:16:00.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:00.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:00 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:16:00.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:00 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:16:01.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:01.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:01.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:01.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:01.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:01.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:01.611 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:01.611 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:01.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:01.679 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:01.680 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:01.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:01 smithi086 ceph-mon[29229]: pgmap v1469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:01.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:01 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:16:01.680 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:01 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:16:01.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:01 smithi117 ceph-mon[32835]: pgmap v1469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:01 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:16:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:01 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:16:01.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:01 smithi107 ceph-mon[32381]: pgmap v1469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:01 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:16:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:01 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:16:02.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:02.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:02.216 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:02.216 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":493,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:15:56.567949+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:02.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:02.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:02.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:02.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:02.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:02.568 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:16:02.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:02.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:02.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1522288296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:02.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:02.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:02.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1522288296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:02.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1522288296' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:03.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:03.224 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:03.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:03.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:03.607 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:03.607 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:03.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:03.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:03.814 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:03 smithi117 ceph-mon[32835]: pgmap v1470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:03.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:03.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:03 smithi086 ceph-mon[29229]: pgmap v1470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:03.918 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:03.918 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:03 smithi107 ceph-mon[32381]: pgmap v1470: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:04.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:04.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:04.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:04.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:04.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:04.589 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:04.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:16:04.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:04.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:04.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:05.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:05.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:05.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:05.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:05.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:05.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:05.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:05.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:05.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:05.689 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:05.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:05.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:05.846 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:16:05.847+0000 7f4114a62640 0 --2- 172.21.15.86:0/2694723420 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f41101018c0 0x7f4110198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:16:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:05 smithi117 ceph-mon[32835]: pgmap v1471: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:05.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:05.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:05 smithi086 ceph-mon[29229]: pgmap v1471: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:05 smithi107 ceph-mon[32381]: pgmap v1471: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:06.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:06.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:06.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:06.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:06.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:06.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:06.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:06.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:06.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:06.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:06.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:06.625 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:16:06.691 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:06.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:06.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:06.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:06.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:06.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:06.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:06.995 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:07.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:07.116 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:07.117 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:07.117 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:07.135 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:07.136 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:07.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:07.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:07.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:07.154 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:07.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:07.155 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:07.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:07.262 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:07.262 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:07.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:07.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:07.543 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:07.543 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:07.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:07.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:07.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:07.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: pgmap v1472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: osdmap e494: 12 total, 3 up, 11 in 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1177647274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:07.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: pgmap v1472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: osdmap e494: 12 total, 3 up, 11 in 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1177647274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:07.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: pgmap v1472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2694723420' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: osdmap e494: 12 total, 3 up, 11 in 2024-08-06T18:16:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1177647274' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:08.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:08.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:08.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:08.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:08.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:08.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:08.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:08.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:08.743 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:08.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:08.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:08.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:08.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:08.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:09.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:09.244 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:09.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:09.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:09.306 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:09.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:09.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:09.546 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:09.547 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:09.547 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:09.547 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:09.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:16:09.653 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:09.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:09.814 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:09 smithi117 ceph-mon[32835]: pgmap v1474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:09.817 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:09 smithi086 ceph-mon[29229]: pgmap v1474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:09.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:09 smithi107 ceph-mon[32381]: pgmap v1474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:10.004 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:10.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:10.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:10.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:10.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:10.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:10.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:10.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:10.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:10.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:16:10.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:10.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:11.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:11.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:11.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:16:11.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:11.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:11.552 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:11.553 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:11.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:11.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:11.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:11.953 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:11 smithi117 ceph-mon[32835]: pgmap v1475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:11.954 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:11 smithi117 ceph-mon[32835]: osdmap e495: 12 total, 3 up, 11 in 2024-08-06T18:16:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:11 smithi107 ceph-mon[32381]: pgmap v1475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:11 smithi107 ceph-mon[32381]: osdmap e495: 12 total, 3 up, 11 in 2024-08-06T18:16:11.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:11 smithi086 ceph-mon[29229]: pgmap v1475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:11.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:11 smithi086 ceph-mon[29229]: osdmap e495: 12 total, 3 up, 11 in 2024-08-06T18:16:11.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:11.977 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:12.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:12.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:12.213 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:12.213 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:12.224 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:12.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:12.379 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:12.379 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":495,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:11.572279+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:12.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:12.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:12.532 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:12.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:12.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1298622830' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:12.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1298622830' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1298622830' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:13.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:13.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:13.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:13.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:13.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:13.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:13.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:13 smithi107 ceph-mon[32381]: pgmap v1477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:13.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:13.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:13 smithi086 ceph-mon[29229]: pgmap v1477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:13 smithi117 ceph-mon[32835]: pgmap v1477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:13.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:14.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:14.221 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:14.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:14.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:14.592 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:14.592 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:14.592 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:14.592 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:14.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:14.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:14.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:15.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:15.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:15.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:15.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:15.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:15.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:15.522 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:15.522 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:15.522 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:15.523 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:15.537 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:15.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:15.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:15.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:15.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:15.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:15.722 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:15 smithi117 ceph-mon[32835]: pgmap v1478: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:15.722 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3760141945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:15.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:15 smithi086 ceph-mon[29229]: pgmap v1478: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:15.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3760141945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:15 smithi107 ceph-mon[32381]: pgmap v1478: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3760141945' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:15.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:16.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:16.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:16.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:16.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:16.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:16.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:16.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:16.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:16.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:16.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:16.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:17.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:17.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:17.196 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:17.196 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:17.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:17.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:17.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:17.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:17.747 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:17 smithi107 ceph-mon[32381]: pgmap v1479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:17.765 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:17.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:17.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:17 smithi086 ceph-mon[29229]: pgmap v1479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:17.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:17.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:17.980 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:16:17.980 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:16:17.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:17 smithi117 ceph-mon[32835]: pgmap v1479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:18.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:18.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:18.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:18.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:18.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:16:18.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:18.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:18.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:19.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:19.103 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:19.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:19.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:19.265 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:19.274 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:19.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:19.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:19.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:19.648 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:19.648 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:19.658 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:16:19.665 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:19.665 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:19.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:19.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:16:19.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:19.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:19 smithi107 ceph-mon[32381]: pgmap v1480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:19.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:20.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:19 smithi086 ceph-mon[29229]: pgmap v1480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:20.001 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:20.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:19 smithi117 ceph-mon[32835]: pgmap v1480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:20.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:20.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:20.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:20.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:20.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:20.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:20.668 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:20.669 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:20.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:20.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:20.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:20 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:20.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:20.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:20 smithi086 ceph-mon[29229]: osdmap e496: 12 total, 3 up, 11 in 2024-08-06T18:16:20.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:20 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:20 smithi107 ceph-mon[32381]: osdmap e496: 12 total, 3 up, 11 in 2024-08-06T18:16:21.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:20 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/776072481' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:20 smithi117 ceph-mon[32835]: osdmap e496: 12 total, 3 up, 11 in 2024-08-06T18:16:21.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:21.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:21.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:21.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:21.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:21.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:21.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:21.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:21.861 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:21.884 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:21 smithi107 ceph-mon[32381]: pgmap v1482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:21 smithi107 ceph-mon[32381]: osdmap e497: 12 total, 3 up, 11 in 2024-08-06T18:16:21.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:21.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:21.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:21 smithi086 ceph-mon[29229]: pgmap v1482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:21.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:21 smithi086 ceph-mon[29229]: osdmap e497: 12 total, 3 up, 11 in 2024-08-06T18:16:21.984 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:21.984 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:22.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:21 smithi117 ceph-mon[32835]: pgmap v1482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:21 smithi117 ceph-mon[32835]: osdmap e497: 12 total, 3 up, 11 in 2024-08-06T18:16:22.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:22.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:22.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:22.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:22.626 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:16:22.626 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:16:22.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:22.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:16:22.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:22.961 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:23.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:23.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:23.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:23.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:23.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:23.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:23.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:23.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:23 smithi107 ceph-mon[32381]: pgmap v1484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:23.932 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:23.933 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:24.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:24.064 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:23 smithi086 ceph-mon[29229]: pgmap v1484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:24.064 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:24.064 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:24.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:23 smithi117 ceph-mon[32835]: pgmap v1484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:24.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:24.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:24.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:24.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:16:24.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:24.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:24.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:24.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:25.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:25.225 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:25.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:25.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:25.417 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:25.417 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":497,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:21.574806+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:02.291405+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:25.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:25.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:25.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:25 smithi117 ceph-mon[32835]: pgmap v1485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/225879865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:25.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:25.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:25 smithi086 ceph-mon[29229]: pgmap v1485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:25.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/225879865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:25 smithi107 ceph-mon[32381]: pgmap v1485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/225879865' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:26.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:26.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:26.194 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:26.194 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:26.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:26.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:26.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:26.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:26.801 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:26.802 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:26.802 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:26.820 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:26.820 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:26.821 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:26.829 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:26.838 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:26.839 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:26.840 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:26.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:26.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:27.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:27.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:27.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:27.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:27.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:27.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:27.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:27.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:27.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:27.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:27 smithi086 ceph-mon[29229]: pgmap v1486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:27.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:27 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:27.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3793823090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:27.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:27.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:27 smithi107 ceph-mon[32381]: pgmap v1486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:27 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3793823090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:27.989 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:27.989 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:28.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:27 smithi117 ceph-mon[32835]: pgmap v1486: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:27 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3793823090' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:28.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:28.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:28.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:28.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:28.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:28.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:28.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:28.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:28.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:29.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:29.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:29.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:29.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:29.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:29.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:29.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:29.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:29.692 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:16:29.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:29.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:29 smithi086 ceph-mon[29229]: pgmap v1487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:29.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:29 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:16:29.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:29 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:16:29.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:29 smithi086 ceph-mon[29229]: osdmap e498: 12 total, 3 up, 11 in 2024-08-06T18:16:29.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:29.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:29.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:29.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:29 smithi107 ceph-mon[32381]: pgmap v1487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:29 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:16:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:29 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:16:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:29 smithi107 ceph-mon[32381]: osdmap e498: 12 total, 3 up, 11 in 2024-08-06T18:16:29.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:29.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:16:30.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:29 smithi117 ceph-mon[32835]: pgmap v1487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:29 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:16:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:29 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:16:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:29 smithi117 ceph-mon[32835]: osdmap e498: 12 total, 3 up, 11 in 2024-08-06T18:16:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:30.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:30.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:30.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:30.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:30.593 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:30.593 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:30.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:30.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:30 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:30.748 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:30.749 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:30 smithi086 ceph-mon[29229]: osdmap e499: 12 total, 3 up, 11 in 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:441,from:0,reservation_nonce:0)", 2024-08-06T18:16:30.834 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 254.21424850099999, 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000182225, 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621109+0000", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0210000000000001e-06 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621113+0000", 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3170000000000003e-06 2024-08-06T18:16:30.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621125+0000", 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1841999999999999e-05 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621133+0000", 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4069999999999993e-06 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621178+0000", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5102000000000003e-05 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621191+0000", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.276e-05 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621290+0000", 2024-08-06T18:16:30.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8776000000000004e-05 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 254.21360509499999, 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010299800000000001, 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.839 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621752+0000", 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5099999999999999e-07 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621762+0000", 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0379e-05 2024-08-06T18:16:30.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621772+0000", 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4450000000000002e-06 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621777+0000", 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.977e-06 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621817+0000", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9956999999999997e-05 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621827+0000", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0652999999999999e-05 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621854+0000", 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6936000000000001e-05 2024-08-06T18:16:30.843 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:457,from:0,reservation_nonce:0)", 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 172.482354122, 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021816399999999999, 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.844 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353003+0000", 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8599999999999998e-07 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353006+0000", 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0249999999999998e-06 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353015+0000", 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9740000000000008e-06 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353020+0000", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2449999999999998e-06 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353061+0000", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1219e-05 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353079+0000", 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7915999999999999e-05 2024-08-06T18:16:30.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353220+0000", 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141099 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 172.48164736199999, 2024-08-06T18:16:30.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6941000000000007e-05, 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:16:30.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353710+0000", 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8500000000000001e-07 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353714+0000", 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8810000000000004e-06 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353724+0000", 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353728+0000", 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3309999999999997e-06 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353769+0000", 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0890000000000003e-05 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353777+0000", 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9880000000000004e-06 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353796+0000", 2024-08-06T18:16:30.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9003000000000001e-05 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:476,from:0,reservation_nonce:0)", 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 106.41410640399999, 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018068000000000001, 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.853 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421251+0000", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8000000000000005e-07 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421254+0000", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9890000000000001e-06 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421263+0000", 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3430000000000002e-06 2024-08-06T18:16:30.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421269+0000", 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8270000000000003e-06 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421316+0000", 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7132000000000002e-05 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421328+0000", 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2038999999999999e-05 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421431+0000", 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010257 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 106.413434448, 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4305e-05, 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:16:30.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.13e-07 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421926+0000", 2024-08-06T18:16:30.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421934+0000", 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6200000000000005e-06 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421941+0000", 2024-08-06T18:16:30.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6229999999999999e-06 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421977+0000", 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5834999999999998e-05 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421986+0000", 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422006+0000", 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0046000000000001e-05 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:16:30.863 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:16:30.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:30 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:30 smithi117 ceph-mon[32835]: osdmap e499: 12 total, 3 up, 11 in 2024-08-06T18:16:30.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:30 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:30.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/416405079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:30.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:30 smithi107 ceph-mon[32381]: osdmap e499: 12 total, 3 up, 11 in 2024-08-06T18:16:30.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:31.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:31.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:31.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:31.161 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:31.161 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:31.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:31.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:31.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:31.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:31.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:31 smithi086 ceph-mon[29229]: pgmap v1490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:31.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:31.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:31.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:31 smithi107 ceph-mon[32381]: pgmap v1490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:32.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:32.099 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:31 smithi117 ceph-mon[32835]: pgmap v1490: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:32.209 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:32.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:32.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:32.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:32.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:16:32.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:32.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:32.636 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:32.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:32.794 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:32.798 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:32.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:33.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:33.028 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:33.028 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:33.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:33.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:33.331 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:33.332 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:33.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:33.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:33.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:16:33.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:33.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:33.824 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:33 smithi107 ceph-mon[32381]: pgmap v1491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:33.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:33 smithi086 ceph-mon[29229]: pgmap v1491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:33.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:34.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:34.076 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:34.076 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:33 smithi117 ceph-mon[32835]: pgmap v1491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:34.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:34.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:34.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:34.306 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:34.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:34.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:34.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:34.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:34.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:34.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:34.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:34.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:34.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:35.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:35.044 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:35.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:16:35.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:35.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:35.294 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:35.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:35.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:35.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:35.685 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:35.685 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:35.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:35.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:35.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:35 smithi086 ceph-mon[29229]: pgmap v1492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:35 smithi107 ceph-mon[32381]: pgmap v1492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:36.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:36.089 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:35 smithi117 ceph-mon[32835]: pgmap v1492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:36.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:36.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:36.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:36.592 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:36.592 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":499,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:29.685938+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:36.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:36.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3534799730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:36.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/98911759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:36.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3534799730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/98911759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:36.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:36.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:36.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3534799730' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:36.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/98911759' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:36.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:37.012 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:37.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:37.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:37.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:37.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:37.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:37.616 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:37.616 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:37.617 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:37.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:37.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:37.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:37.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:37.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:37.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:37.728 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:37.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:16:37.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:37.816 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:37 smithi086 ceph-mon[29229]: pgmap v1493: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:37.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:37 smithi117 ceph-mon[32835]: pgmap v1493: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:37.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.959 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:37 smithi107 ceph-mon[32381]: pgmap v1493: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:37.977 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:38.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:38.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:38.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:38.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:38.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:38.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:38.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:39.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:39.134 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:39.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:39.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:39.441 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:39.441 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:39.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:39.630 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:39.631 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:39.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.694 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:39.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.695 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:39 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.751 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:39.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:16:39.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:16:39.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:39.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.826 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:39 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:16:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:16:39.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:16:39.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:39 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:16:39.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:40.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:40.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:40.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:40.358 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:40.359 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:40.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:40.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:16:40.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:40.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:40.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:40.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:40.696 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:40.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:40.797 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:40.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:40.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:41.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:41.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:41.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:41.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:41.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:41.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:41.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:41.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:41.617 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:16:41.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:16:41.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:41.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:41.826 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:41 smithi107 ceph-mon[32381]: pgmap v1495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:41.826 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:16:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:41 smithi086 ceph-mon[29229]: pgmap v1495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:16:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:41 smithi117 ceph-mon[32835]: pgmap v1495: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:16:41.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:42.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:42.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:42.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:42.441 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:42.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:42.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:42.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:42.747 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:42.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:42 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:16:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:42 smithi086 ceph-mon[29229]: osdmap e500: 12 total, 3 up, 11 in 2024-08-06T18:16:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:42 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:16:42.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:42 smithi117 ceph-mon[32835]: osdmap e500: 12 total, 3 up, 11 in 2024-08-06T18:16:42.887 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:42 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:42.888 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1825347284' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:16:42.888 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:42 smithi107 ceph-mon[32381]: osdmap e500: 12 total, 3 up, 11 in 2024-08-06T18:16:42.888 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:42.888 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:42.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:43.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:43.163 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:43.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:16:43.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:43.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:43.412 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:43.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:43.623 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:43.623 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":500,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:41.611596+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:43.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:43 smithi107 ceph-mon[32381]: pgmap v1497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:43.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:43.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:43.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:43 smithi086 ceph-mon[29229]: pgmap v1497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:43 smithi117 ceph-mon[32835]: pgmap v1497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:43.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:44.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:44.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:44.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:44.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:44.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:44.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:16:44.726 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:44.757 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1520344546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:44.757 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:44.757 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:44.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:44.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1520344546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:44.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:44.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1520344546' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:44.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:45.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:45.158 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:45.158 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:45.197 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:45.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:45.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:45.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:45.418 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:45.423 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:45.423 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:45.520 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:45.521 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:45.521 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:45.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:45.538 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:45.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:45.540 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:45.549 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:45.557 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:45.558 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:45.559 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:45.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:45.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:45 smithi107 ceph-mon[32381]: pgmap v1498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:45.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/242720335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:45.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:45.814 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:45 smithi086 ceph-mon[29229]: pgmap v1498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:45.815 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/242720335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:45.837 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:45.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:16:45.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:45.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:45 smithi117 ceph-mon[32835]: pgmap v1498: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:45.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/242720335' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:46.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:46.087 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:46.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:46.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:46.225 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:46.225 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:46.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:46.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:46.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:46.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:46.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:46 smithi086 ceph-mon[29229]: osdmap e501: 12 total, 3 up, 11 in 2024-08-06T18:16:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:46 smithi117 ceph-mon[32835]: osdmap e501: 12 total, 3 up, 11 in 2024-08-06T18:16:46.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:46.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:46 smithi107 ceph-mon[32381]: osdmap e501: 12 total, 3 up, 11 in 2024-08-06T18:16:46.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:47.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:47.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:47.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:47.313 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:47.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:47.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:47.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:47.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:47.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:47.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:47.728 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:47.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:47.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:47 smithi086 ceph-mon[29229]: pgmap v1499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:47 smithi117 ceph-mon[32835]: pgmap v1499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:47.897 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:47.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:47 smithi107 ceph-mon[32381]: pgmap v1499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:48.053 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:48.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:48.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:16:48.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:48.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:48.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:48.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:48.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:16:48.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:48.637 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:16:48.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:48.795 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:48.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:48.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:48.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:49.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:16:49.135 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:49.135 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:49.135 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:49.135 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:49.244 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:49.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:49.512 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:49.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:49 smithi107 ceph-mon[32381]: pgmap v1501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:49 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:49 smithi107 ceph-mon[32381]: osdmap e502: 12 total, 3 up, 11 in 2024-08-06T18:16:49.773 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:49.779 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:49 smithi117 ceph-mon[32835]: pgmap v1501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:49 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:49.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:49 smithi117 ceph-mon[32835]: osdmap e502: 12 total, 3 up, 11 in 2024-08-06T18:16:49.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:49 smithi086 ceph-mon[29229]: pgmap v1501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:49.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:49 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:16:49.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2280801753' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:16:49.892 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:49 smithi086 ceph-mon[29229]: osdmap e502: 12 total, 3 up, 11 in 2024-08-06T18:16:49.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:50.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:50.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:50.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:50.410 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:50.410 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:50.433 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:16:50.433 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:16:50.433 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:16:50.433 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:16:50.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:50.601 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:50.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:50.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:51.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:51.088 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:16:51.088 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":502,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:48.635342+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:37.188228+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:16:51.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:16:51.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:51.270 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:51.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:51.343 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:51.343 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:51.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:16:51.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:16:51.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:16:51.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:51.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:16:51.780 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:51 smithi107 ceph-mon[32381]: pgmap v1503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:51.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3103011287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:51.781 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:51 smithi107 ceph-mon[32381]: osdmap e503: 12 total, 3 up, 11 in 2024-08-06T18:16:51.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:51.910 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:51.910 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:51.910 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:51 smithi086 ceph-mon[29229]: pgmap v1503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:51.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3103011287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:51.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:51 smithi086 ceph-mon[29229]: osdmap e503: 12 total, 3 up, 11 in 2024-08-06T18:16:51.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:52.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:52.122 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:52.123 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:52.123 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:51 smithi117 ceph-mon[32835]: pgmap v1503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3103011287' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:16:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:51 smithi117 ceph-mon[32835]: osdmap e503: 12 total, 3 up, 11 in 2024-08-06T18:16:52.140 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:52.141 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:52.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:52.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:52.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:52.159 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:52.159 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:52.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:52.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:52.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:52.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:52.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:52.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:52.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:52.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:52.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:52.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:52.727 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:52.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1714233890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:52.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:52.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:52.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1714233890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:52.917 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:52.995 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:53.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1714233890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:53.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:53.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:53.073 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:53.073 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:53.087 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:16:53.088 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:16:53.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:53.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:53.368 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:16:53.369 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:16:53.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:53.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:53.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:53.694 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:53 smithi117 ceph-mon[32835]: pgmap v1505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:53.694 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:53.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:16:53.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:53.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:53 smithi086 ceph-mon[29229]: pgmap v1505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:53.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:53.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:53.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:53.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:53.954 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:53.956 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:53.957 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:53.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:53 smithi107 ceph-mon[32381]: pgmap v1505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:53.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:54.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:54.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:54.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:16:54.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:54.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:54.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:54.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:55.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:55.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:55.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:55.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:16:55.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:55.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:55.614 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:55.655 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:16:55.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:55.670 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:16:55.670 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:16:55.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:16:55.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:55 smithi117 ceph-mon[32835]: pgmap v1506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:16:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:16:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:55 smithi117 ceph-mon[32835]: osdmap e504: 12 total, 3 up, 11 in 2024-08-06T18:16:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:55.918 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:55 smithi086 ceph-mon[29229]: pgmap v1506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:55.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:16:55.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:16:55.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:55 smithi086 ceph-mon[29229]: osdmap e504: 12 total, 3 up, 11 in 2024-08-06T18:16:55.919 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:55.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:55.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:55 smithi107 ceph-mon[32381]: pgmap v1506: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:55.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:16:55.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:16:55.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:55 smithi107 ceph-mon[32381]: osdmap e504: 12 total, 3 up, 11 in 2024-08-06T18:16:55.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:16:56.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:56.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:56.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:56.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:16:56.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:56.472 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:56.473 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:56.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:56.701 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:56.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:56.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:56 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:56.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:56.823 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:56 smithi107 ceph-mon[32381]: osdmap e505: 12 total, 3 up, 11 in 2024-08-06T18:16:56.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:56 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:56.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:56 smithi117 ceph-mon[32835]: osdmap e505: 12 total, 3 up, 11 in 2024-08-06T18:16:56.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:57.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:56 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:16:57.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3699763225' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:16:57.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:56 smithi086 ceph-mon[29229]: osdmap e505: 12 total, 3 up, 11 in 2024-08-06T18:16:57.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:57.041 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:57.077 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:57.077 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:57.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:57.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:57.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:57.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:57.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:57.659 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:16:57.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:16:57.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:57 smithi107 ceph-mon[32381]: pgmap v1509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:57.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:16:57.939 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:57 smithi117 ceph-mon[32835]: pgmap v1509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:58.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:57 smithi086 ceph-mon[29229]: pgmap v1509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:58.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:58.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:16:58.309 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:16:58.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:58.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:58.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:58.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:16:58.777 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:16:58.777 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:16:58.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:16:58.793 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:16:58.794 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:16:58.794 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:16:58.794 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:16:58.794 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:16:58.812 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:16:58.813 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:16:58.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:58.943 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:16:58.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:16:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:16:58.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:16:59.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:16:59.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:59.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:16:59.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:59.265 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:59.316 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:16:59.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:16:59.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:16:59.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:16:59.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:16:59.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:16:59.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:16:59.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:59 smithi086 ceph-mon[29229]: pgmap v1510: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:59.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3258713342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:59.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:59.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:16:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:59.936 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:59 smithi117 ceph-mon[32835]: pgmap v1510: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:59.936 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3258713342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:59.936 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:59.936 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:16:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:59.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:16:59.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:16:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:16:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:59 smithi107 ceph-mon[32381]: pgmap v1510: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:16:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3258713342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:16:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:16:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:16:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:00.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:00.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:00.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:00.346 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:00.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:00.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:00.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:00.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:00.728 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:00.762 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:00.795 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:00.795 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":505,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:16:55.655291+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:15:44.198718+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:00.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:00.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:00.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:00.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:00.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:00.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:00.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:00.906 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:00.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:01.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:01.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:01.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:01.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:01.510 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:01.511 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:17:01.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:01.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:01.823 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:17:01.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:01.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:01 smithi117 ceph-mon[32835]: pgmap v1511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:01.927 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/112326848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:01.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:01 smithi086 ceph-mon[29229]: pgmap v1511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:01.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/112326848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:01.934 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:01.934 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:01.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:01 smithi107 ceph-mon[32381]: pgmap v1511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:01.935 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/112326848' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:02.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:02.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:02.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:02.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:17:02.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:02.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:02.687 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:02.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:02.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:02.936 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:02.937 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:02.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:03.092 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:03.187 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:03.187 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:03.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:03.343 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:03.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:03.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:03.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:03.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:03 smithi117 ceph-mon[32835]: pgmap v1512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:03.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:03.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:03 smithi107 ceph-mon[32381]: pgmap v1512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:03.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:03 smithi086 ceph-mon[29229]: pgmap v1512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:03.964 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:03.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:17:04.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:04.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:04.220 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:04.287 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:04.417 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:04.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:04.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:04.684 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:17:04.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:04.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:04.934 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:04.934 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:04.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:04.935 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:04.935 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:04.936 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:04.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:04.952 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:04.952 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:04.952 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:04.954 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:04.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:05.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:05.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:05.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:05.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:05.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:05.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:05.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:05.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:05.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:05.766 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:05.767 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:05.851 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: pgmap v1513: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: osdmap e506: 12 total, 3 up, 11 in 2024-08-06T18:17:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1591529099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:05.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:05.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:05.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: pgmap v1513: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: osdmap e506: 12 total, 3 up, 11 in 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1591529099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:06.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: pgmap v1513: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4126269124' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: osdmap e506: 12 total, 3 up, 11 in 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1591529099' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:06.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:06.350 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:06.351 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:06.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:06.417 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:06.467 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:06.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:17:06.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:06.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:06.769 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:06.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:06.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:06 smithi107 ceph-mon[32381]: osdmap e507: 12 total, 3 up, 11 in 2024-08-06T18:17:06.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.971 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:06 smithi086 ceph-mon[29229]: osdmap e507: 12 total, 3 up, 11 in 2024-08-06T18:17:06.992 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:06.993 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:06 smithi117 ceph-mon[32835]: osdmap e507: 12 total, 3 up, 11 in 2024-08-06T18:17:07.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:07.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:07.237 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:07.237 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:07.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:07.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:07.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:07.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:07.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:17:07.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:07.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:07 smithi117 ceph-mon[32835]: pgmap v1515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:07.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:07 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:17:07.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:07 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:17:07.945 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:07.945 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:07.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:07 smithi086 ceph-mon[29229]: pgmap v1515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:07.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:07 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:17:07.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:07 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:17:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:07 smithi107 ceph-mon[32381]: pgmap v1515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:07 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:17:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:07 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:17:08.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:08.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:08.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:08.304 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:08.306 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:08.378 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:08.378 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":507,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:06.586661+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:08.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:17:08.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:08.450 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:08.450 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:08.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:08.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:08.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/180694887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:08.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/180694887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:08.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/180694887' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:08.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:08.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:09.004 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:09.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:17:09.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:09.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:09.288 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:09.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:09.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:09.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:09.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:09.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:09.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:09.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:09.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:09.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:09 smithi086 ceph-mon[29229]: pgmap v1517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:09.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:09.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:09.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:09.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:09 smithi117 ceph-mon[32835]: pgmap v1517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:09 smithi107 ceph-mon[32381]: pgmap v1517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:09.974 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:10.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:10.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:10.160 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:10.160 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:10.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:10.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:10.475 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:10.475 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:10.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:10.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:10.766 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:10.766 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:10.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:10.923 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:10.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:11.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:11.198 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:11.198 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:11.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:11.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:11.213 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:11.213 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:11.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:11.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:11.214 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:11.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:11.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:11.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:11.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:11.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:11.513 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:11.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:17:11.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:11.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:11.746 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:11 smithi117 ceph-mon[32835]: pgmap v1518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:11.746 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1735909328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:11.746 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:11.799 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:11.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:11.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:11.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:11 smithi086 ceph-mon[29229]: pgmap v1518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1735909328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:11 smithi107 ceph-mon[32381]: pgmap v1518: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1735909328' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:11.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:12.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:12.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:12.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:12.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:12.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:12.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:12.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:12.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:12.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:12.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:12.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:12.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:12.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:13.016 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:13.016 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:13.043 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:13.045 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:13.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:13.245 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:13.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:13.335 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:13.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:13.495 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:13.495 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:13.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:13.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:13.708 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:17:13.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:13 smithi086 ceph-mon[29229]: pgmap v1519: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:13.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:13.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:13 smithi117 ceph-mon[32835]: pgmap v1519: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:13.935 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:13.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:17:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:13 smithi107 ceph-mon[32381]: pgmap v1519: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:13.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:14.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:14.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:14.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:14.183 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:14.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:14.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:14.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:14.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:14.670 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:14.671 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:14.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:14 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:14 smithi117 ceph-mon[32835]: osdmap e508: 12 total, 3 up, 11 in 2024-08-06T18:17:14.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:14 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:14 smithi107 ceph-mon[32381]: osdmap e508: 12 total, 3 up, 11 in 2024-08-06T18:17:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:15.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:15.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:17:15.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:14 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:15.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1316333668' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:15.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:14 smithi086 ceph-mon[29229]: osdmap e508: 12 total, 3 up, 11 in 2024-08-06T18:17:15.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:15.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:15.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:15.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:15.667 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:15.667 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:15.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:15.784 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:15.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:15.941 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:15.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:15 smithi086 ceph-mon[29229]: pgmap v1521: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:15.941 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":508,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:13.704662+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:15.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:15 smithi107 ceph-mon[32381]: pgmap v1521: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:15.972 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:15 smithi117 ceph-mon[32835]: pgmap v1521: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:16.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:16.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:16.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:16.381 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:16.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:17:16.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:16.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:16.627 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:16.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:16.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:17:16.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/633117500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:16.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:16 smithi086 ceph-mon[29229]: osdmap e509: 12 total, 3 up, 11 in 2024-08-06T18:17:16.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:16.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/633117500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:16.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:16 smithi117 ceph-mon[32835]: osdmap e509: 12 total, 3 up, 11 in 2024-08-06T18:17:17.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:17.041 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/633117500' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:17.041 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:16 smithi107 ceph-mon[32381]: osdmap e509: 12 total, 3 up, 11 in 2024-08-06T18:17:17.041 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:17.041 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:17.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:17.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:17.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:17.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:17.666 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:17.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:17.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:17.835 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:17 smithi107 ceph-mon[32381]: pgmap v1522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:17 smithi117 ceph-mon[32835]: pgmap v1522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:17.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:17 smithi086 ceph-mon[29229]: pgmap v1522: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:17.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:18.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:18.173 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:18.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:18.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:18.474 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:18.474 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:18.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:18.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:18.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:18.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:18.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:18.769 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:18 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:17:18.769 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:18 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:17:18.818 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:18.819 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:18.819 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:18.835 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:18.836 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:18.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:18.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:18.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:18.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:18.854 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:18.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:18 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:17:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:18 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:17:18.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:18.923 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:18.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:17:18.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:18 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:17:18.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:18 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:17:19.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:19.140 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:19.207 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:19.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:19.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:19.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:19.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:19.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:19.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:19.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:19.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:19.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:19 smithi086 ceph-mon[29229]: pgmap v1524: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:19.841 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1651646273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:19.841 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:19.841 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:19 smithi117 ceph-mon[32835]: pgmap v1524: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1651646273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:19.920 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:19.920 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:19.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:20.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:20.073 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:19 smithi107 ceph-mon[32381]: pgmap v1524: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:20.073 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1651646273' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:20.073 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:20.073 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:20.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:20.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:20.336 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:20.336 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:20.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:20.489 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:20.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:20.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:20.728 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:20.729 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:17:20.843 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:20.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:20.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:20.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:21.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:21.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:21.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:21.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:21.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:21.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:21.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:21.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:17:21.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:21.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:21.739 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:21.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:21.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:21.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:21.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:21.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:21.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:21 smithi086 ceph-mon[29229]: pgmap v1525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:21.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:21 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:21.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:21.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:21 smithi086 ceph-mon[29229]: osdmap e510: 12 total, 3 up, 11 in 2024-08-06T18:17:21.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:21 smithi086 ceph-mon[29229]: osdmap e511: 12 total, 3 up, 11 in 2024-08-06T18:17:21.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:21.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:17:22.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:22.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:21 smithi117 ceph-mon[32835]: pgmap v1525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:22.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:21 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:22.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:22.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:21 smithi117 ceph-mon[32835]: osdmap e510: 12 total, 3 up, 11 in 2024-08-06T18:17:22.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:21 smithi117 ceph-mon[32835]: osdmap e511: 12 total, 3 up, 11 in 2024-08-06T18:17:22.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:21 smithi107 ceph-mon[32381]: pgmap v1525: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:22.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:21 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:22.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2960363851' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:22.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:21 smithi107 ceph-mon[32381]: osdmap e510: 12 total, 3 up, 11 in 2024-08-06T18:17:22.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:21 smithi107 ceph-mon[32381]: osdmap e511: 12 total, 3 up, 11 in 2024-08-06T18:17:22.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:22.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:22.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:22.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:22.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:22.712 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:22.712 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:22.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:22.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:22.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:22.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:22.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:23.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:23.131 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:23.131 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":511,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:21.590904+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:23.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:23.273 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:23.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:23.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:23.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:23.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:23.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:23.733 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:23.733 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:23.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:23.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:23.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:23 smithi107 ceph-mon[32381]: pgmap v1528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:23.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3487789024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:23.954 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:23.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:17:24.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:23 smithi086 ceph-mon[29229]: pgmap v1528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:24.023 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3487789024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:24.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:24.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:23 smithi117 ceph-mon[32835]: pgmap v1528: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:24.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3487789024' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:24.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:24.197 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:24.197 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:24.203 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:24.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:24.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:24.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:17:24.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:24.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:24.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:24.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:24.772 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:24.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:24.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:24.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:24.973 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:25.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:25.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:25.230 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:25.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:25.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:25.466 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:25.466 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:25.467 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:25.480 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:25.481 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:25.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:25.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:25.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:25.498 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:25.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:25.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:25.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:25.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:25.729 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:25.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:25.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:25 smithi117 ceph-mon[32835]: pgmap v1529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:25.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1552032836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:25.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:25 smithi107 ceph-mon[32381]: pgmap v1529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1552032836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:25.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:26.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:25 smithi086 ceph-mon[29229]: pgmap v1529: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1552032836' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:26.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:26.299 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:26.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:17:26.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:26.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:26.546 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:26.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:26.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:26.590 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:26.699 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:26.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:26.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:26.991 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:27.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:27.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:27.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:27.255 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:27.255 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:27.364 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:27.389 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:27.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:27.523 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:27.523 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:27.524 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:27.524 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:27.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:27.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:27.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:27.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:27.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:27 smithi086 ceph-mon[29229]: pgmap v1530: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:27.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:27.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:27 smithi117 ceph-mon[32835]: pgmap v1530: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:27.938 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:28.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:28.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:28.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:27 smithi107 ceph-mon[32381]: pgmap v1530: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:28.192 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:28.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:28.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:28.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:28.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:28.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:28.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:28.752 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:17:28.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:28.863 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:28.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:17:28.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:28.969 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:17:29.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:29.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:17:29.108 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:17:29.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:29.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:29.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:29.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:29.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:17:29.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:29.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:29.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:29.753 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:29.753 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:29.753 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:29 smithi107 ceph-mon[32381]: pgmap v1531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:29.753 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:29 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:29.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:29 smithi117 ceph-mon[32835]: pgmap v1531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:29 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:17:29.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:29 smithi117 ceph-mon[32835]: osdmap e512: 12 total, 3 up, 11 in 2024-08-06T18:17:29.982 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:29.982 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:29.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:29 smithi086 ceph-mon[29229]: pgmap v1531: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:29.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:29 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:29.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:17:29.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:29 smithi086 ceph-mon[29229]: osdmap e512: 12 total, 3 up, 11 in 2024-08-06T18:17:29.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:30.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1331802097' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:17:30.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:29 smithi107 ceph-mon[32381]: osdmap e512: 12 total, 3 up, 11 in 2024-08-06T18:17:30.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:30.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:30.339 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:30.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:30.511 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:30.511 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":512,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:28.749207+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:28.671010+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:30.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:30.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:30.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:30.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:30.723 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:30.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:30.811 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2088791086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:30.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:30.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2088791086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:30.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2088791086' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:31.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:31.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:31.256 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:31.256 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:31.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:31.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:31.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:17:31.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:31.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:31.633 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:17:31.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:31.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:17:31.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:31.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:31 smithi086 ceph-mon[29229]: pgmap v1533: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:31.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:31 smithi086 ceph-mon[29229]: osdmap e513: 12 total, 3 up, 11 in 2024-08-06T18:17:31.864 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:31 smithi117 ceph-mon[32835]: pgmap v1533: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:31.865 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:31 smithi117 ceph-mon[32835]: osdmap e513: 12 total, 3 up, 11 in 2024-08-06T18:17:31.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:32.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:31 smithi107 ceph-mon[32381]: pgmap v1533: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:32.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:31 smithi107 ceph-mon[32381]: osdmap e513: 12 total, 3 up, 11 in 2024-08-06T18:17:32.129 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:32.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:32.299 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:32.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:32.461 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:32.461 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:32.461 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:32.461 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:32.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:32.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:32.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:32.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:32.967 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:32.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:32.984 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:32.984 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:32.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:32.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:32.985 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:32.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:33.002 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:33.003 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:33.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:33.202 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:33.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:33.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:33.410 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:33.410 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:33.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:33.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:33.759 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:33.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:17:33.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:33.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:33 smithi086 ceph-mon[29229]: pgmap v1535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:33.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1103977886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:33.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:33.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:33.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:33 smithi107 ceph-mon[32381]: pgmap v1535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1103977886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:33.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:34.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:34.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:33 smithi117 ceph-mon[32835]: pgmap v1535: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:34.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1103977886' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:34.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:34.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:34.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:34.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:34.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:34.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:34.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:34.544 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:34.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:34.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:34.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:34.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:34.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:34.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:17:34.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:17:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:17:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:34 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:17:34.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:35.062 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:35.062 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:35.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:35.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:17:35.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:17:35.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:17:35.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:34 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:17:35.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:17:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:17:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:17:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:34 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:17:35.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:35.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:35.316 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:35.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:35.379 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:35.379 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:35.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:35.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:35.768 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:17:35.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:35.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:35 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:17:35.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:35 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:17:35.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:35 smithi086 ceph-mon[29229]: osdmap e514: 12 total, 3 up, 11 in 2024-08-06T18:17:35.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:35.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:35 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:17:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:35 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:17:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:35 smithi107 ceph-mon[32381]: osdmap e514: 12 total, 3 up, 11 in 2024-08-06T18:17:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:36.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:36.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:36.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:35 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:17:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:35 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:17:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:35 smithi117 ceph-mon[32835]: osdmap e514: 12 total, 3 up, 11 in 2024-08-06T18:17:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:36.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:36.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:36.395 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:36.395 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:36.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:36.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:36.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:36.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:36.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:36.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:36.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:37.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:37.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:37.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:36 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:37.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:36 smithi086 ceph-mon[29229]: osdmap e515: 12 total, 3 up, 11 in 2024-08-06T18:17:37.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:36 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:37.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:37.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:36 smithi107 ceph-mon[32381]: osdmap e515: 12 total, 3 up, 11 in 2024-08-06T18:17:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:36 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4245340086' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:36 smithi117 ceph-mon[32835]: osdmap e515: 12 total, 3 up, 11 in 2024-08-06T18:17:37.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:37.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:17:37.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:37.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:37.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:37.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:37 smithi117 ceph-mon[32835]: pgmap v1539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:37.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:37 smithi107 ceph-mon[32381]: pgmap v1539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:38.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:38.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:37 smithi086 ceph-mon[29229]: pgmap v1539: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:38.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:38.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:38.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:38.599 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:38.599 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:38.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:38.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:38.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:38.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:38.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:38.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:38 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:38.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:39.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:38 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:39.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:38 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:39.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:39.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:39.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:39.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:39.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:39.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:39.690 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:39.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:39.857 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:39.858 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":515,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:35.769377+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:34.466918+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:39.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:39 smithi107 ceph-mon[32381]: pgmap v1540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:39.966 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:39 smithi117 ceph-mon[32835]: pgmap v1540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:40.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:39 smithi086 ceph-mon[29229]: pgmap v1540: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:40.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:40.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:40.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:40.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:40.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:17:40.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:40.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:40.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2236271191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:40.909 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:40.909 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:41.018 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:41.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2236271191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:41.159 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:41.159 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:41.160 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:41.160 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:41.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:41.191 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2236271191' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:41.300 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:41.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:41.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:41.691 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:41.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:41.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:41.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:41.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:41.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:42.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: pgmap v1541: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:17:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:17:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: pgmap v1541: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:17:42.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: pgmap v1541: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:17:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:17:42.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:42.557 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:42.557 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:42.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:42.771 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:42.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:43.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:43.117 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:43.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:17:43.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:43.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:43.339 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:43.354 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:43.355 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:43.355 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:43.377 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:43.377 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:43.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:43.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:43.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:43.395 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:43.396 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:43.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:43.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:43.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:43.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:43.397 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:43.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:43.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:43.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:43.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:43.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:43.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:44.011 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:43 smithi107 ceph-mon[32381]: pgmap v1542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:44.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1921292874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:44.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:43 smithi086 ceph-mon[29229]: pgmap v1542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1921292874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:44.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:44.147 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:43 smithi117 ceph-mon[32835]: pgmap v1542: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:44.148 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1921292874' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:44.148 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:44.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:44.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:44.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:44.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:44.729 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:44.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:44.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:44.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:44.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:45.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:45.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:45.230 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:45.230 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:45.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:45.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:45.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:45.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:45.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:46.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:46.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:45 smithi086 ceph-mon[29229]: pgmap v1543: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:46.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:46.169 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:45 smithi117 ceph-mon[32835]: pgmap v1543: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:46.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:45 smithi107 ceph-mon[32381]: pgmap v1543: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:46.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:46.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:46.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:46.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:46.694 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:46.720 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:46.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:46.839 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:46.839 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:46.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:46.956 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:17:47.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:47.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:47.217 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:47.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:47.227 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:47.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:17:47.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:47.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:47.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:47.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:47.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:47.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:48.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:47 smithi117 ceph-mon[32835]: pgmap v1544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:47 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:47 smithi117 ceph-mon[32835]: osdmap e516: 12 total, 3 up, 11 in 2024-08-06T18:17:48.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:48.175 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:48.175 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:48.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:47 smithi086 ceph-mon[29229]: pgmap v1544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:48.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:47 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:48.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:48.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:47 smithi086 ceph-mon[29229]: osdmap e516: 12 total, 3 up, 11 in 2024-08-06T18:17:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:47 smithi107 ceph-mon[32381]: pgmap v1544: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:47 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:17:48.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3356912956' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:17:48.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:47 smithi107 ceph-mon[32381]: osdmap e516: 12 total, 3 up, 11 in 2024-08-06T18:17:48.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:48.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:48.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:48.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:48.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:17:48.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:48.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:48.954 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:48.954 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:49.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:49.095 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:49.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:17:49.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:48 smithi107 ceph-mon[32381]: pgmap v1546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:49.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:48 smithi117 ceph-mon[32835]: pgmap v1546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:49.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:48 smithi086 ceph-mon[29229]: pgmap v1546: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:49.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:49.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:49.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:49.688 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:49.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:49.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:49.901 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:49.902 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:49.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:17:50.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:50.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:50.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:50.417 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:50.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:50.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:50.816 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:50.889 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:17:50.890 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":516,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:17:46.945979+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:34.466918+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:17:50.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:51.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:51.159 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:51.159 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:51.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:51.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:51 smithi117 ceph-mon[32835]: pgmap v1547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2496249058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:51.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:51 smithi107 ceph-mon[32381]: pgmap v1547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:51.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2496249058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:51.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:51.510 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:51.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:51 smithi086 ceph-mon[29229]: pgmap v1547: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:51.598 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2496249058' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:17:51.599 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:51.599 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:51.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:51.744 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:51.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:51.979 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:17:51.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:17:51.980 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:17:51.980 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:17:51.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:17:51.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:17:51.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:17:51.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:17:51.982 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:17:51.983 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:17:51.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:17:51.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:17:51.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:17:51.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:17:51.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:17:52.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:52.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:52.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:52.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:52.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:52.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:52.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:52.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:52.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:52 smithi107 ceph-mon[32381]: osdmap e517: 12 total, 3 up, 11 in 2024-08-06T18:17:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/332585788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:52.838 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:52.839 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:52.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:52 smithi086 ceph-mon[29229]: osdmap e517: 12 total, 3 up, 11 in 2024-08-06T18:17:52.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/332585788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:52.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:52 smithi117 ceph-mon[32835]: osdmap e517: 12 total, 3 up, 11 in 2024-08-06T18:17:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/332585788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:17:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:52.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:53.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:53.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:53.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:53.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:53.289 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:17:53.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:17:53.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:53.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:53.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:53.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:53.507 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:53.507 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:53.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:53.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:53.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:53.791 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:53 smithi086 ceph-mon[29229]: pgmap v1549: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:53.869 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:53 smithi117 ceph-mon[32835]: pgmap v1549: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:53.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:53.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:17:53.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:53 smithi107 ceph-mon[32381]: pgmap v1549: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:53.911 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:53.911 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:54.020 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:54.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:17:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:54.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:54.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:54.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:54.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:54.707 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:54 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:17:54.707 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:54 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:17:54.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:54.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:54 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:17:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:54 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:17:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:54 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:17:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:54 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:17:54.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:55.001 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:17:55.088 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:17:55.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:55.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:17:55.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:55.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:55.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:55.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:55 smithi086 ceph-mon[29229]: pgmap v1550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:55.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:55 smithi117 ceph-mon[32835]: pgmap v1550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:55.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:55 smithi107 ceph-mon[32381]: pgmap v1550: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:56.020 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:56.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:56.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:56.314 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:56.315 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:56.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:56.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:56.635 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:17:56.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:17:56.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:56.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:17:56.839 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:56.839 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:56.839 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:56.839 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:56.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:56.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:57.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:57.208 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:57.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:57.280 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:57.280 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:57.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:57.526 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:57.619 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:17:57.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:57.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:57.861 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:57 smithi086 ceph-mon[29229]: pgmap v1551: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:57.861 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:57 smithi117 ceph-mon[32835]: pgmap v1551: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:57.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:17:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:57 smithi107 ceph-mon[32381]: pgmap v1551: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:17:58.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:58.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:17:58.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:17:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:17:58.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:58.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:17:58.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:58.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:58.616 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:58.622 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:17:58.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:58.777 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:58 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:58.777 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:58.777 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:58 smithi107 ceph-mon[32381]: osdmap e518: 12 total, 3 up, 11 in 2024-08-06T18:17:58.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:58.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:58 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:58.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:58 smithi086 ceph-mon[29229]: osdmap e518: 12 total, 3 up, 11 in 2024-08-06T18:17:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:58 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:17:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2537564451' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:17:58.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:58 smithi117 ceph-mon[32835]: osdmap e518: 12 total, 3 up, 11 in 2024-08-06T18:17:58.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:17:59.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:59.045 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:17:59.045 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:17:59.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:17:59.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:17:59.308 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:17:59.308 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:17:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:17:59.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:17:59.532 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:17:59.532 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:17:59.556 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:17:59.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:17:59.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:17:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:17:59 smithi117 ceph-mon[32835]: pgmap v1553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:17:59.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:17:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:17:59 smithi107 ceph-mon[32381]: pgmap v1553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:00.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:17:59 smithi086 ceph-mon[29229]: pgmap v1553: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:00.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:18:00.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:00.255 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:00.255 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:17:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:00.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:00.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:00.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:18:00.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:00.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:00.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:00.983 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:01.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:01.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:01.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:01.106 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:01.106 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:01.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:01.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:01.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:01.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:01.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:01.571 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:01.571 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:01.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:01.719 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:01 smithi086 ceph-mon[29229]: pgmap v1554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:01.720 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:01 smithi086 ceph-mon[29229]: osdmap e519: 12 total, 3 up, 11 in 2024-08-06T18:18:01.720 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:01.720 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:01.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:01.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:01 smithi107 ceph-mon[32381]: pgmap v1554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:01.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:01 smithi107 ceph-mon[32381]: osdmap e519: 12 total, 3 up, 11 in 2024-08-06T18:18:01.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:01.864 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:01.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:02.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:01 smithi117 ceph-mon[32835]: pgmap v1554: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:01 smithi117 ceph-mon[32835]: osdmap e519: 12 total, 3 up, 11 in 2024-08-06T18:18:02.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:02.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:02.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:02.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:02.397 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:02.397 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:02.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:02.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:02.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:02.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:02.855 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":519,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:01.600147+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:34.466918+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:02.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:03.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:03.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:03.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:03.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:03.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:03.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:03.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:03 smithi117 ceph-mon[32835]: pgmap v1556: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2178112329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:03 smithi107 ceph-mon[32381]: pgmap v1556: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2178112329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:03.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:03 smithi086 ceph-mon[29229]: pgmap v1556: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2178112329' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:04.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:04.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:04.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:04.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:04.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:04.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:04.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:04.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:04.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:05.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:05.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:05.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:05.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:05.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:05.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:05.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:05.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:05.758 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:05 smithi086 ceph-mon[29229]: pgmap v1557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:05.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:05.892 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:18:05.892 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:18:05.902 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:05 smithi107 ceph-mon[32381]: pgmap v1557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:06.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:06.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:06.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:05 smithi117 ceph-mon[32835]: pgmap v1557: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:06.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:06.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:06.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:06.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:06.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:06.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:06.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:06.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:06.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:06.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:06.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:06.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:06.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:07.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:07.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:18:07.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:07.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:07.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:07.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:18:07.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:07.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:07.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:07.897 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:07.897 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:07.897 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:07 smithi086 ceph-mon[29229]: pgmap v1558: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:07.906 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:07 smithi107 ceph-mon[32381]: pgmap v1558: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:08.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:08.042 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:07 smithi117 ceph-mon[32835]: pgmap v1558: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:08.050 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:08.050 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:08.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:08.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:08.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:08.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:08.655 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:18:08.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:08.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:08.771 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:08.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:08.852 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:08.870 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:08.870 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:08.871 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:08.887 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:08.888 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:08.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:08.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:08.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:08.906 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:08.906 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:08.906 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:08.906 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:08.906 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:08.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:08.907 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:08.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:09.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:09.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:09.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:09.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:09.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:09.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:09.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:09.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:09.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:09.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:09.803 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:09.803 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:09.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: pgmap v1559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:09.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:09.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:09.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: osdmap e520: 12 total, 3 up, 11 in 2024-08-06T18:18:09.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1323736973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:09.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:09.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:09.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: pgmap v1559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: osdmap e520: 12 total, 3 up, 11 in 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1323736973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:09.921 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:10.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: pgmap v1559: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2932641555' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: osdmap e520: 12 total, 3 up, 11 in 2024-08-06T18:18:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1323736973' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:10.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:10.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:10.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:10.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:10.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:10.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:10.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:10.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:10.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:10.994 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:11.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:11.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:11.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:11.392 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:11.393 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:11.393 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:11.393 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:11.393 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:11.394 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:11.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:11.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:11.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:11.809 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:11 smithi107 ceph-mon[32381]: pgmap v1561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:11.809 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:11 smithi107 ceph-mon[32381]: osdmap e521: 12 total, 3 up, 11 in 2024-08-06T18:18:11.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:11.945 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:11 smithi117 ceph-mon[32835]: pgmap v1561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:11.945 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:11 smithi117 ceph-mon[32835]: osdmap e521: 12 total, 3 up, 11 in 2024-08-06T18:18:11.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:11 smithi086 ceph-mon[29229]: pgmap v1561: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:11.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:11 smithi086 ceph-mon[29229]: osdmap e521: 12 total, 3 up, 11 in 2024-08-06T18:18:12.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:12.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:12.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:12.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:12.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:12.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:12.481 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:12.481 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:12.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:18:12.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:387,from:6,reservation_nonce:15)", 2024-08-06T18:18:12.627 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:18:12.628 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 565.40722150199997, 2024-08-06T18:18:12.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000152874, 2024-08-06T18:18:12.628 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.628 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.628 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552878+0000", 2024-08-06T18:18:12.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552879+0000", 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.579e-06 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552889+0000", 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6840000000000008e-06 2024-08-06T18:18:12.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552910+0000", 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1277e-05 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552922+0000", 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1421e-05 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.552990+0000", 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8114999999999994e-05 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553003+0000", 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3453e-05 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553031+0000", 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7345e-05 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.632 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:387/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 565.40642920699997, 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000125494, 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553670+0000", 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553671+0000", 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4799999999999999e-07 2024-08-06T18:18:12.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553674+0000", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8e-06 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553701+0000", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6699999999999998e-05 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553708+0000", 2024-08-06T18:18:12.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0559999999999999e-06 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553764+0000", 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5810000000000003e-05 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553772+0000", 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.5560000000000002e-06 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.553796+0000", 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4124000000000001e-05 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:387,from:6,reservation_nonce:0)", 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 565.40564424800004, 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011286299999999999, 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.637 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554455+0000", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554456+0000", 2024-08-06T18:18:12.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6100000000000002e-07 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554458+0000", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.582e-06 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554463+0000", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0520000000000001e-06 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554471+0000", 2024-08-06T18:18:12.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7999999999999999e-06 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554549+0000", 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8207000000000005e-05 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554559+0000", 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.023e-05 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:08:46.554568+0000", 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6309999999999992e-06 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:423,from:6,reservation_nonce:17)", 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 412.94122244300002, 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014934700000000001, 2024-08-06T18:18:12.641 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:18:12.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018879+0000", 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1619999999999998e-06 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018888+0000", 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1570000000000006e-06 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018906+0000", 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7773000000000001e-05 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018917+0000", 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1262e-05 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018984+0000", 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6549000000000003e-05 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018999+0000", 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5593000000000001e-05 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019026+0000", 2024-08-06T18:18:12.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6851000000000001e-05 2024-08-06T18:18:12.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.646 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:423/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 412.94039327000002, 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014629899999999999, 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.647 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019707+0000", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6499999999999999e-07 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019740+0000", 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3853999999999998e-05 2024-08-06T18:18:12.648 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019750+0000", 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019761+0000", 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1636e-05 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.649 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019809+0000", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019820+0000", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1606999999999999e-05 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019852+0000", 2024-08-06T18:18:12.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2240000000000003e-05 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:423,from:6,reservation_nonce:0)", 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 412.93943747499998, 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000163024, 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.651 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.86e-07 2024-08-06T18:18:12.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020664+0000", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.579e-06 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020685+0000", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1350999999999999e-05 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020689+0000", 2024-08-06T18:18:12.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2910000000000001e-06 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020805+0000", 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011613 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020816+0000", 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.048e-05 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020825+0000", 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0070000000000003e-06 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:441,from:2,reservation_nonce:16)", 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 355.34018256500002, 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011107860000000001, 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.655 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619918+0000", 2024-08-06T18:18:12.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619927+0000", 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619942+0000", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5214999999999999e-05 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619950+0000", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9519999999999994e-06 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620000+0000", 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9979999999999999e-05 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620004+0000", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0589999999999996e-06 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620978+0000", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097387000000000005 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620993+0000", 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4676e-05 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621028+0000", 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4950999999999998e-05 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:441/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.661 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 355.338575225, 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5687999999999997e-05, 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621524+0000", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621525+0000", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8700000000000005e-07 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621530+0000", 2024-08-06T18:18:12.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3519999999999997e-06 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621541+0000", 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0828e-05 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621546+0000", 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.383e-06 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621585+0000", 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8364000000000001e-05 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621591+0000", 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9649999999999996e-06 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621610+0000", 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9108999999999999e-05 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:441,from:2,reservation_nonce:0)", 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 355.33793894899998, 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111129, 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622161+0000", 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7599999999999998e-07 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622163+0000", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.951e-06 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622167+0000", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8650000000000003e-06 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622170+0000", 2024-08-06T18:18:12.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5260000000000002e-06 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622240+0000", 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9622000000000005e-05 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622257+0000", 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7754e-05 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622272+0000", 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4134999999999999e-05 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:457,from:2,reservation_nonce:17)", 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 273.60726300099998, 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010675699999999999, 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.670 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352837+0000", 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7300000000000005e-07 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352842+0000", 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2769999999999999e-06 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352852+0000", 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352859+0000", 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6959999999999999e-06 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352906+0000", 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7364999999999998e-05 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352919+0000", 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3159999999999999e-05 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352943+0000", 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3828e-05 2024-08-06T18:18:12.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:457/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 273.60662320500001, 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3411000000000004e-05, 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353477+0000", 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4199999999999995e-07 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353481+0000", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5820000000000003e-06 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353493+0000", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1887e-05 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353499+0000", 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6010000000000002e-06 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353543+0000", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3927000000000001e-05 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353549+0000", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5549999999999999e-06 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353570+0000", 2024-08-06T18:18:12.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0217e-05 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:457,from:2,reservation_nonce:0)", 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 273.60590287000002, 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012299899999999999, 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.39e-07 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354199+0000", 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5150000000000001e-06 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354206+0000", 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0090000000000001e-06 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354219+0000", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2714999999999999e-05 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354300+0000", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1031000000000004e-05 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354310+0000", 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0359e-05 2024-08-06T18:18:12.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354320+0000", 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0310000000000004e-06 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:476,from:2,reservation_nonce:18)", 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 207.53904203900001, 2024-08-06T18:18:12.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012406, 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421059+0000", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.471e-06 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421067+0000", 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0600000000000008e-06 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421082+0000", 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305e-05 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421093+0000", 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421142+0000", 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8767999999999998e-05 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421156+0000", 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4027e-05 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421181+0000", 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5871e-05 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:476/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 207.53844088700001, 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119988, 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4099999999999999e-07 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421663+0000", 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3649999999999997e-06 2024-08-06T18:18:12.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421673+0000", 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0149000000000001e-05 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421681+0000", 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9920000000000007e-06 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421726+0000", 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4246999999999998e-05 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421737+0000", 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1796e-05 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421778+0000", 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0998000000000003e-05 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:476,from:2,reservation_nonce:0)", 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 207.53777746899999, 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111465, 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.692 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:18:12.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2300000000000002e-07 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422325+0000", 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422331+0000", 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2530000000000001e-06 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422336+0000", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.558e-06 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422406+0000", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9059000000000004e-05 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422420+0000", 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422433+0000", 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3499000000000001e-05 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:509,from:2,reservation_nonce:20)", 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.799104945000003, 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011134249999999999, 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160996+0000", 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161004+0000", 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2330000000000006e-06 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161020+0000", 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5559000000000001e-05 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161028+0000", 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2819999999999996e-06 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161088+0000", 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9743e-05 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161092+0000", 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7160000000000002e-06 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162061+0000", 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00096807299999999996 2024-08-06T18:18:12.700 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162076+0000", 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5928999999999999e-05 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162108+0000", 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1380000000000001e-05 2024-08-06T18:18:12.701 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:509/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.797519674999997, 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112194, 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.702 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:18:12.703 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8400000000000004e-07 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162592+0000", 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1977000000000001e-05 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162605+0000", 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2602000000000001e-05 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.704 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162614+0000", 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.918e-06 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162663+0000", 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9168000000000001e-05 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162670+0000", 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.396e-06 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.705 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162692+0000", 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1449000000000001e-05 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:509,from:2,reservation_nonce:0)", 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 54.796875806999999, 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112296, 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:12.706 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.707 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9499999999999998e-07 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163226+0000", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1940000000000001e-06 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163231+0000", 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1819999999999997e-06 2024-08-06T18:18:12.708 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163235+0000", 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8290000000000001e-06 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163305+0000", 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0358000000000006e-05 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.709 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163322+0000", 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6384000000000001e-05 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163336+0000", 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4054e-05 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.710 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.711 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.711 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:12.711 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:12.711 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:18:12.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:12.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:12.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:13.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:13.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:13.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:13.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:13.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:13.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:18:13.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:13.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:13.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:13.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:13.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:13.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:13 smithi086 ceph-mon[29229]: pgmap v1563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:13.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:18:13.921 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:13.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:13.952 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:13 smithi117 ceph-mon[32835]: pgmap v1563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:13 smithi107 ceph-mon[32381]: pgmap v1563: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 325 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:14.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:14.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:14.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:14.456 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:14.456 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:14.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:14.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:14.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:14.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:14.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:14.766 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:14.767 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:14.841 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:14.841 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":521,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:11.602955+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:16:54.305190+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:34.466918+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:14.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:15.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:15.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:15.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:15.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:15.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:15.662 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:15.663 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:15.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:15.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:15 smithi107 ceph-mon[32381]: pgmap v1564: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:15.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1672352008' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:15.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:15.835 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:15.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:16.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:15 smithi086 ceph-mon[29229]: pgmap v1564: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:16.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1672352008' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:16.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:16.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:15 smithi117 ceph-mon[32835]: pgmap v1564: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:16.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1672352008' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:16.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:16.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:16.363 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:16.363 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:16.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:16.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:16.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:16.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:16.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:17.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:17.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:17.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:17.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:17.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:17.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:17.711 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:17 smithi117 ceph-mon[32835]: pgmap v1565: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:17.819 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:17 smithi086 ceph-mon[29229]: pgmap v1565: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:17.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:17.845 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:17 smithi107 ceph-mon[32381]: pgmap v1565: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:17.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:18.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:18.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:18.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:18.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:18.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:18.672 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:18.672 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:18.679 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:18:18.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:18.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:18:18.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:18.921 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:19.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:19.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:19.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:19.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:19.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:19.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:19.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:19.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:19 smithi086 ceph-mon[29229]: pgmap v1566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:19.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:19 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:19.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:19.735 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:19 smithi086 ceph-mon[29229]: osdmap e522: 12 total, 3 up, 11 in 2024-08-06T18:18:19.739 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:18:19.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:19 smithi117 ceph-mon[32835]: pgmap v1566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:19 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:19 smithi117 ceph-mon[32835]: osdmap e522: 12 total, 3 up, 11 in 2024-08-06T18:18:19.920 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:19 smithi107 ceph-mon[32381]: pgmap v1566: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:19 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1453665521' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:19.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:19 smithi107 ceph-mon[32381]: osdmap e522: 12 total, 3 up, 11 in 2024-08-06T18:18:19.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:20.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:20.156 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:20.157 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:20.157 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:20.173 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:20.174 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:20.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:20.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:20.174 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:20.192 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:20.192 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:20.192 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:20.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:20.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:20.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:20.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:20.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:20.271 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:20.271 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:20.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:20.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:20.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:20.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:20.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2934371655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:20.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:20.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:20.943 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:18:20.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:18:20.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2934371655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:20.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:20.968 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2934371655' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:20.968 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:20.968 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:21.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:21.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:21.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:21.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:21.577 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:21.577 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:21.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:21.607 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:21.607 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:21.736 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:21.845 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:21.846 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:21.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:21 smithi086 ceph-mon[29229]: pgmap v1568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:21.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:21.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:18:21.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:18:21.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:21 smithi086 ceph-mon[29229]: osdmap e523: 12 total, 3 up, 11 in 2024-08-06T18:18:21.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:21.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:21 smithi117 ceph-mon[32835]: pgmap v1568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:21.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:21.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:18:21.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:18:21.893 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:21 smithi117 ceph-mon[32835]: osdmap e523: 12 total, 3 up, 11 in 2024-08-06T18:18:21.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:21 smithi107 ceph-mon[32381]: pgmap v1568: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:18:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:18:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:21 smithi107 ceph-mon[32381]: osdmap e523: 12 total, 3 up, 11 in 2024-08-06T18:18:22.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:22.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:22.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:22.400 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:22.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:22.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:22.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:22.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:22.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:22.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:23.037 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:23.062 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:23.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:23.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:23.467 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:23.521 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:23.521 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:23.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:23.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:18:23.683 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:23.683 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:23.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:23.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:23.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:23 smithi086 ceph-mon[29229]: pgmap v1570: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:23.867 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:23 smithi107 ceph-mon[32381]: pgmap v1570: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:24.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:23 smithi117 ceph-mon[32835]: pgmap v1570: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:24.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:24.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:24.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:24.318 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:24.319 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":523,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:20.676488+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:34.466918+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:17:06.327151+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:24.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:24.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:24.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:24.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:24.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:18:24.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:24.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:24.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/40759075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:24.831 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:24.831 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/40759075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:24.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:24.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/40759075' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:25.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:25.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:25.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:25.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:25.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:25.435 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:25.435 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:25.484 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:25.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:25.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:25.782 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:25 smithi117 ceph-mon[32835]: pgmap v1571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:25.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:25 smithi107 ceph-mon[32381]: pgmap v1571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:25.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:25 smithi086 ceph-mon[29229]: pgmap v1571: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:26.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:26.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:26.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:26.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:26.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:26.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:26.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:26.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:26.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:26.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:27.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:27.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:27.036 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:18:27.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:27.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:27.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:27.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:27.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:27.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:27.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:27.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:27.673 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:27.673 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:27.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:27.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:27.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:27 smithi086 ceph-mon[29229]: pgmap v1572: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:27.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:27 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:18:27.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:27 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:18:27.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:27 smithi086 ceph-mon[29229]: osdmap e524: 12 total, 3 up, 11 in 2024-08-06T18:18:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:18:27.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:18:27.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:27 smithi117 ceph-mon[32835]: pgmap v1572: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:27.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:27 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:18:27.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:27 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:18:27.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:27 smithi117 ceph-mon[32835]: osdmap e524: 12 total, 3 up, 11 in 2024-08-06T18:18:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:27 smithi107 ceph-mon[32381]: pgmap v1572: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:27 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:18:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:27 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:18:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:27 smithi107 ceph-mon[32381]: osdmap e524: 12 total, 3 up, 11 in 2024-08-06T18:18:28.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:28.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:28.257 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:28.257 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:28.257 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:28.272 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:28.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:28.280 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:28.280 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:28.280 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:28.281 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:28.281 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:28.307 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:28.307 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:28.307 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:28.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:28.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:28.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:28.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:28.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:28.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:28.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:28.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:28.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:28.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2897563721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:28.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:28.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:28.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:28.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:28.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2897563721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:28.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:29.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:18:29.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2897563721' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:29.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:29.245 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:29.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:29.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:29.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:29.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:29.689 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:29.689 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:29.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:29.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:29 smithi117 ceph-mon[32835]: pgmap v1574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:29.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:29.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:29.923 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:29.924 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:29.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:29.961 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:29.961 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:29.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:29 smithi107 ceph-mon[32381]: pgmap v1574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:29.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:29.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:30.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:29 smithi086 ceph-mon[29229]: pgmap v1574: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:30.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:30.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:30.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:30.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:30.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:30.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:30.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:30.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:30.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:30.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:30.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:30.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:30.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:31.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:31.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:31.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:31.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:31.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:31.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:31.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:31.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:31 smithi086 ceph-mon[29229]: pgmap v1575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:31.825 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:31 smithi086 ceph-mon[29229]: osdmap e525: 12 total, 3 up, 11 in 2024-08-06T18:18:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:31 smithi117 ceph-mon[32835]: pgmap v1575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:31 smithi117 ceph-mon[32835]: osdmap e525: 12 total, 3 up, 11 in 2024-08-06T18:18:31.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:18:31.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:31 smithi107 ceph-mon[32381]: pgmap v1575: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:31 smithi107 ceph-mon[32381]: osdmap e525: 12 total, 3 up, 11 in 2024-08-06T18:18:32.036 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:18:32.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:32.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:32.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:32.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:32.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:32.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:32.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:32.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:32.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:32.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:32.828 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:32.829 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:32.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:33.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:33.129 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:33.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:18:33.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:18:33.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:33.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:33.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:33.454 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:33.454 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:33.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:33.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:33.714 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:18:33.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:33 smithi107 ceph-mon[32381]: pgmap v1577: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:18:34.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:34.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:33 smithi117 ceph-mon[32835]: pgmap v1577: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:34.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:18:34.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:33 smithi086 ceph-mon[29229]: pgmap v1577: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:34.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:18:34.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:34.278 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:34.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:34.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:34.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:34.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:34.553 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:34.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:34.815 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:34 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:18:34.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:34 smithi086 ceph-mon[29229]: osdmap e526: 12 total, 3 up, 11 in 2024-08-06T18:18:34.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:34.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:34.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:34 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:18:34.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:34 smithi107 ceph-mon[32381]: osdmap e526: 12 total, 3 up, 11 in 2024-08-06T18:18:34.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:34 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:34.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2642951602' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:18:34.977 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:34 smithi117 ceph-mon[32835]: osdmap e526: 12 total, 3 up, 11 in 2024-08-06T18:18:35.016 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:35.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:35.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:35.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:35.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:35.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:35.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:35.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:35.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:35.773 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:35 smithi086 ceph-mon[29229]: pgmap v1579: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:35.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:36.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:36.038 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:35 smithi117 ceph-mon[32835]: pgmap v1579: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:36.084 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:36.084 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:36.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:36.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:36.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:35 smithi107 ceph-mon[32381]: pgmap v1579: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:36.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:36.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:36.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:36.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:36.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:18:36.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:36.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:36.696 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:36.696 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:36.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:36 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:18:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:36 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:18:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:36 smithi117 ceph-mon[32835]: osdmap e527: 12 total, 3 up, 11 in 2024-08-06T18:18:36.892 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:36.893 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":527,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:35.721303+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:36.958 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:36.958 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:36.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:36 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:18:36.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:36 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:18:36.958 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:36 smithi086 ceph-mon[29229]: osdmap e527: 12 total, 3 up, 11 in 2024-08-06T18:18:36.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:37.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:37.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:36 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:18:37.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:36 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:18:37.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:36 smithi107 ceph-mon[32381]: osdmap e527: 12 total, 3 up, 11 in 2024-08-06T18:18:37.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:37.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:37.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:37.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:37.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:37.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:37.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:37.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:37.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:37 smithi086 ceph-mon[29229]: pgmap v1581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:37.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/739942953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:37.896 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:37.896 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:37.912 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:37 smithi117 ceph-mon[32835]: pgmap v1581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:37.912 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/739942953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:38.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:38.114 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:38.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:37 smithi107 ceph-mon[32381]: pgmap v1581: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/739942953' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:38.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:38.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:38.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:38.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:38.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:38.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:39.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:39.119 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:39.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:39.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:39.135 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:39.136 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:39.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:39.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:39.136 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:39.153 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:39.154 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:39.155 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:39.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:39.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:39.489 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:39.623 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:39.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:39.783 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:39 smithi117 ceph-mon[32835]: pgmap v1582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:39.783 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1652080471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:39.783 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:39.784 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:39.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:39.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:40.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:40.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:39 smithi086 ceph-mon[29229]: pgmap v1582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:40.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1652080471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:40.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:39 smithi107 ceph-mon[32381]: pgmap v1582: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1652080471' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:40.332 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:40.332 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:40.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:40.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:40.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:40.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:40.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:18:40.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:40.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:40.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:40 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:18:40.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:40 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:18:40.874 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:18:40.875 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:18:40.962 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:40.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:40.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:40 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:18:40.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:40 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:18:41.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:41.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:41.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:40 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:18:41.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:40 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:18:41.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:41.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:41.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:41.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:18:41.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:41.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:41.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:41 smithi086 ceph-mon[29229]: pgmap v1583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:41 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:18:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:41.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:41 smithi117 ceph-mon[32835]: pgmap v1583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:41 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:18:41.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:41.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:18:42.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:42.060 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:41 smithi107 ceph-mon[32381]: pgmap v1583: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:42.060 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:41 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:18:42.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:42.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:42.191 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:42.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:42.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:42.447 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:42.448 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:42.448 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:42.448 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:42.540 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:42.563 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:42.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:42.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:42.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:43.033 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:43.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:43.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:43.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:43.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:43.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:43.747 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:43.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:43.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: pgmap v1584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:43.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:43.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:43.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:43.901 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:18:43.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:18:43.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: pgmap v1584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:18:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:18:43.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:44.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:44.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: pgmap v1584: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:44.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:44.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:44.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:18:44.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:18:44.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:18:44.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:18:44.062 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:18:44.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:44.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:44.296 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:44.297 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:44.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:44.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:44.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:44.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:44.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:44.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:44 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:18:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:44.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:44 smithi086 ceph-mon[29229]: osdmap e528: 12 total, 3 up, 11 in 2024-08-06T18:18:44.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:44 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:18:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:44 smithi107 ceph-mon[32381]: osdmap e528: 12 total, 3 up, 11 in 2024-08-06T18:18:45.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:44 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:18:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2933726484' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:18:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:44 smithi117 ceph-mon[32835]: osdmap e528: 12 total, 3 up, 11 in 2024-08-06T18:18:45.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:45.321 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:45.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:45.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:45.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:45.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:45.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:45.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:45.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:45.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:45 smithi086 ceph-mon[29229]: pgmap v1586: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:46.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:46.040 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:45 smithi107 ceph-mon[32381]: pgmap v1586: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:45 smithi117 ceph-mon[32835]: pgmap v1586: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:46.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:46.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:46.305 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:18:46.305 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:18:46.305 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:46.306 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:46.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:46.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:46.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:46.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:46.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:46.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:46.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:46.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:18:47.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:18:47.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:47.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:47.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:47.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:47.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:47.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:47.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:47 smithi086 ceph-mon[29229]: pgmap v1587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:47 smithi086 ceph-mon[29229]: osdmap e529: 12 total, 3 up, 11 in 2024-08-06T18:18:47.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:47.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:47 smithi117 ceph-mon[32835]: pgmap v1587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:47 smithi117 ceph-mon[32835]: osdmap e529: 12 total, 3 up, 11 in 2024-08-06T18:18:47.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:47 smithi107 ceph-mon[32381]: pgmap v1587: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:47 smithi107 ceph-mon[32381]: osdmap e529: 12 total, 3 up, 11 in 2024-08-06T18:18:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:48.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:48.059 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:48.059 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:48.060 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:48.072 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:48.073 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:48.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:48.074 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:48.074 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:48.095 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:48.097 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:48.097 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:48.097 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:48.098 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:48.098 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:48.098 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:48.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:48.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:48.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:48.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:48.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:48.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:48.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:48.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:48.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:48.755 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:48.755 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:48.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2208168914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:48.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.755 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.842 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2208168914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2208168914' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:48.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:48.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:49.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:49.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:49.251 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:49.251 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":529,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:46.612951+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:49.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:49.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:49.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:49.538 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:49.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:49 smithi107 ceph-mon[32381]: pgmap v1589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:49.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1693683203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:49.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:49.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:49.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:49 smithi086 ceph-mon[29229]: pgmap v1589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:49.807 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1693683203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:49.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:49.807 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:49.831 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:49.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:49.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:50.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:50.085 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:50.086 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:49 smithi117 ceph-mon[32835]: pgmap v1589: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1693683203' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:50.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:50.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:50.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:50.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:50.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:50.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:50.740 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:50.740 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:50.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:50.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:51.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:51.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:51.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:51.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:51.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:51.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:51.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:18:51.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:51.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:51 smithi086 ceph-mon[29229]: pgmap v1590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:51.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:51 smithi117 ceph-mon[32835]: pgmap v1590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:51 smithi107 ceph-mon[32381]: pgmap v1590: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:51.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:52.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:52.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:18:52.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:52.402 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:18:52.403 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:18:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:18:52.503 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:18:52.503 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:441,from:0,reservation_nonce:0)", 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 395.88354388699997, 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000182225, 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.504 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621108+0000", 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.505 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621109+0000", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0210000000000001e-06 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621113+0000", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3170000000000003e-06 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621125+0000", 2024-08-06T18:18:52.506 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1841999999999999e-05 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621133+0000", 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4069999999999993e-06 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621178+0000", 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5102000000000003e-05 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.507 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621191+0000", 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.276e-05 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621290+0000", 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8776000000000004e-05 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.508 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 395.88290048099998, 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010299800000000001, 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.509 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.510 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621751+0000", 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621752+0000", 2024-08-06T18:18:52.511 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5099999999999999e-07 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621762+0000", 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0379e-05 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.512 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621772+0000", 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4450000000000002e-06 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621777+0000", 2024-08-06T18:18:52.513 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.977e-06 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621817+0000", 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.9956999999999997e-05 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621827+0000", 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0652999999999999e-05 2024-08-06T18:18:52.514 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621854+0000", 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6936000000000001e-05 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.515 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:457,from:0,reservation_nonce:0)", 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 314.15164950799999, 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021816399999999999, 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.516 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.517 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353003+0000", 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8599999999999998e-07 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353006+0000", 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0249999999999998e-06 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353015+0000", 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9740000000000008e-06 2024-08-06T18:18:52.518 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353020+0000", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2449999999999998e-06 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353061+0000", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1219e-05 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353079+0000", 2024-08-06T18:18:52.519 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7915999999999999e-05 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353220+0000", 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141099 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.520 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 314.15094274799998, 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6941000000000007e-05, 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.521 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.522 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353710+0000", 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8500000000000001e-07 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.523 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353714+0000", 2024-08-06T18:18:52.524 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8810000000000004e-06 2024-08-06T18:18:52.524 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.524 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:52.525 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.525 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353724+0000", 2024-08-06T18:18:52.525 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-08-06T18:18:52.525 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353728+0000", 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3309999999999997e-06 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353769+0000", 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0890000000000003e-05 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.526 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353777+0000", 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9880000000000004e-06 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353796+0000", 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9003000000000001e-05 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.527 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:476,from:0,reservation_nonce:0)", 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 248.08340179000001, 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018068000000000001, 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.528 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.529 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421251+0000", 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8000000000000005e-07 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421254+0000", 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9890000000000001e-06 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.530 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421263+0000", 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3430000000000002e-06 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421269+0000", 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8270000000000003e-06 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421316+0000", 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7132000000000002e-05 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.531 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421328+0000", 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2038999999999999e-05 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421431+0000", 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010257 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.532 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 248.08272983399999, 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4305e-05, 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:18:52.533 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:18:52.534 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.13e-07 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421926+0000", 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421934+0000", 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6200000000000005e-06 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.535 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421941+0000", 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6229999999999999e-06 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421977+0000", 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5834999999999998e-05 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421986+0000", 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.536 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422006+0000", 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0046000000000001e-05 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:509,from:0,reservation_nonce:0)", 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 95.342453191000004, 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018985599999999999, 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.537 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.538 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162200+0000", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3599999999999999e-06 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162205+0000", 2024-08-06T18:18:52.539 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.938e-06 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162219+0000", 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3974e-05 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162227+0000", 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-06 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.540 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162279+0000", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2927999999999999e-05 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162293+0000", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3737000000000001e-05 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162388+0000", 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5088999999999997e-05 2024-08-06T18:18:52.541 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 95.341812113000003, 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3175000000000003e-05, 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:18:52.542 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:18:52.543 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1900000000000002e-07 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162843+0000", 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3230000000000002e-06 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:18:52.544 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162852+0000", 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5979999999999997e-06 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162858+0000", 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.19e-06 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162889+0000", 2024-08-06T18:18:52.545 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1059000000000002e-05 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162896+0000", 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.635e-06 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:18:52.546 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162913+0000", 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6750999999999999e-05 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:18:52.547 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:18:52.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:52.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:52.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:53.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:53.117 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:53.117 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:53.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:53.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:18:53.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:53.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:53.574 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:18:53.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:18:53.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:53.727 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:53.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:53.781 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:53 smithi086 ceph-mon[29229]: pgmap v1591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:53 smithi117 ceph-mon[32835]: pgmap v1591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:53.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:53.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:53 smithi107 ceph-mon[32381]: pgmap v1591: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:54.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:54.097 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:54.097 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:54.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:54.177 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:54.177 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:54.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:54.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:54.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:54.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:54.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:54.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:54.966 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:54.966 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:55.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:55.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:55.273 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:18:55.279 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:55.279 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:55.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:55.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:18:55.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:55.624 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:18:55.624 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:18:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:18:55.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:55.852 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:55 smithi086 ceph-mon[29229]: pgmap v1592: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:55.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:55.894 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:55 smithi117 ceph-mon[32835]: pgmap v1592: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:55.917 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:55 smithi107 ceph-mon[32381]: pgmap v1592: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:55.944 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:18:55.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:18:56.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:56.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:56.193 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:18:56.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:56.308 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:18:56.339 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:18:56.340 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:18:56.340 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:18:56.361 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:18:56.362 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:18:56.362 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:18:56.362 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:18:56.362 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:18:56.377 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:18:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:18:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:18:56.377 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:18:56.378 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:18:56.378 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:18:56.378 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:18:56.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:56.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:56.652 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:18:56.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:56.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:56.715 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/595274455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:56.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/595274455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:56.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:57.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:18:57.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:18:57.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/595274455' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:18:57.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:57.251 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:57.251 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:57.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:18:57.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:57.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:57.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:57.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:57.735 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:18:57.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:57.787 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: pgmap v1593: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:57.787 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.787 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.787 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:57.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:57.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:57 smithi107 ceph-mon[32381]: osdmap e530: 12 total, 3 up, 11 in 2024-08-06T18:18:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: pgmap v1593: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:57.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:57 smithi117 ceph-mon[32835]: osdmap e530: 12 total, 3 up, 11 in 2024-08-06T18:18:57.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:18:58.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:58.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: pgmap v1593: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1462447889' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:18:58.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:57 smithi086 ceph-mon[29229]: osdmap e530: 12 total, 3 up, 11 in 2024-08-06T18:18:58.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:58.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:58.317 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:58.317 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:58.339 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:18:58.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:18:58.408 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:18:58.408 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:18:58.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:18:58.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:58.578 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:58.578 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:58.592 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:18:58.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:58.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:58.838 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:58.838 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:58.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:59.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:18:59.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:59.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:59.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:18:59.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:59.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:18:59.498 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:18:59.498 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:18:59.582 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:18:59.582 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":530,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:18:56.646420+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:18:59.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:18:59.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:18:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:59 smithi086 ceph-mon[29229]: pgmap v1595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:18:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/472830518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:18:59.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:18:59.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:59 smithi107 ceph-mon[32381]: pgmap v1595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:18:59.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:18:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/472830518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:00.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:00.112 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:59 smithi117 ceph-mon[32835]: pgmap v1595: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:18:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/472830518' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:00.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:00.154 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:00.155 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:18:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:00.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:00.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:00.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:19:00.430 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:00.430 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:00.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:00.654 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:00.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:00.683 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:00.683 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:00.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:19:00.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:00.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:19:00.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:01.018 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:01.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:01.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:01.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:01.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:01.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:01.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:01.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:01.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:01.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:01 smithi107 ceph-mon[32381]: pgmap v1596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:01.788 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:01 smithi107 ceph-mon[32381]: osdmap e531: 12 total, 3 up, 11 in 2024-08-06T18:19:01.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:01 smithi117 ceph-mon[32835]: pgmap v1596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:01 smithi117 ceph-mon[32835]: osdmap e531: 12 total, 3 up, 11 in 2024-08-06T18:19:01.948 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:01.948 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:01.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:01 smithi086 ceph-mon[29229]: pgmap v1596: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:01.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:01 smithi086 ceph-mon[29229]: osdmap e531: 12 total, 3 up, 11 in 2024-08-06T18:19:01.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:02.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:02.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:02.316 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:02.328 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:02.328 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:02.329 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:02.347 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:02.348 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:02.348 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:02.348 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:02.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:02.366 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:02.366 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:02.366 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:02.366 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:02.367 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:02.367 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:02.367 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:02.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:02.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:02.600 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:02.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:02.659 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:02.659 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:02.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:02.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/536784251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:02.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:02.942 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/536784251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/536784251' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:03.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:03.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:03.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:03.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:03.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:19:03.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:03.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:03.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:03.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:03.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:03.503 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:03.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:03.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:03.742 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:03.742 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:03.762 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:03.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:03 smithi117 ceph-mon[32835]: pgmap v1598: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:03 smithi107 ceph-mon[32381]: pgmap v1598: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:03.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.931 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:03.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:03 smithi086 ceph-mon[29229]: pgmap v1598: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:04.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:04.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:04.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:04.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:04.441 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:04.441 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:04.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:04.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:04.682 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:04.682 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:04.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:04.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:04.945 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:04.945 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:05.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:05.077 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:05.077 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:05.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:05.279 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:05.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:05.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:05.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:05.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:05.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:05.599 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:05.599 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:05.599 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:05.600 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:05.669 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:19:05.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:05.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:19:05.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:05.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:05 smithi107 ceph-mon[32381]: pgmap v1599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:05.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:05.958 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:05.959 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:05.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:05 smithi086 ceph-mon[29229]: pgmap v1599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:05.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:05.961 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:05.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:19:05.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:06.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:05 smithi117 ceph-mon[32835]: pgmap v1599: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:06.210 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:06.214 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:06.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:06.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:06.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:06.538 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:06.538 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:06.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:06.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:06.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:06 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:06.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:06.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:06 smithi117 ceph-mon[32835]: osdmap e532: 12 total, 3 up, 11 in 2024-08-06T18:19:06.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:06 smithi117 ceph-mon[32835]: osdmap e533: 12 total, 3 up, 11 in 2024-08-06T18:19:06.854 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:06.855 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:06.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:06 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:06.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:06.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:06 smithi086 ceph-mon[29229]: osdmap e532: 12 total, 3 up, 11 in 2024-08-06T18:19:06.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:06 smithi086 ceph-mon[29229]: osdmap e533: 12 total, 3 up, 11 in 2024-08-06T18:19:06.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:06.917 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:06.918 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":533,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:06.617860+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:06.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:06 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:06.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3243522094' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:06.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:06 smithi107 ceph-mon[32381]: osdmap e532: 12 total, 3 up, 11 in 2024-08-06T18:19:06.933 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:06 smithi107 ceph-mon[32381]: osdmap e533: 12 total, 3 up, 11 in 2024-08-06T18:19:07.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:07.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:07.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:07.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:07.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:07.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:07.440 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:07.440 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:07.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:07.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:07.839 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:07.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:07 smithi086 ceph-mon[29229]: pgmap v1601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:07.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2116541543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:07.859 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:07.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:07 smithi117 ceph-mon[32835]: pgmap v1601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:07.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2116541543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:07 smithi107 ceph-mon[32381]: pgmap v1601: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2116541543' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:07.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:08.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:08.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:08.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:08.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:08.616 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:08.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:19:08.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:08.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:08.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:08.799 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:08.862 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:08.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:09.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:09.052 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:09.052 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:09.053 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:09.053 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:09.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:09.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:09.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:09.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:09.644 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:09.644 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:09.644 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:09.661 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:09.662 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:09.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:09.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:09.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:09.681 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:09.681 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:09.685 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:09.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:09.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:09.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:09.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:09.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:09.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:09.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:19:09.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: pgmap v1603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:09.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1714871611,config=507150784,health=2077816597,kv=1077530148}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {kv=11,logm=89} crc {kv=1169701999,logm=1272414743}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403702368}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=580189836}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309969450}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371969285}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090047580}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106414157}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095974148}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831979699}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984507092}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357188910}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145306464}) 2024-08-06T18:19:09.928 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229625650}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186237441}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=806453732}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808529168}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3634317104}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=483247400}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568927254}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825396583}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421523470}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970339360}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894802967}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195030136}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869429989}) 2024-08-06T18:19:09.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960963129}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249957449}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557524930}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862529299}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112582149}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=943626556}) 2024-08-06T18:19:09.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3155336851}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=682635022}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156461017}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2755819301}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2854687652}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072541837}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339411591}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010561712}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733980372}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108165419}) 2024-08-06T18:19:09.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782450109}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036111200}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965811701}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204950315}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585816328}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446229589}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4126774433}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414517977}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893880449}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901020071}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616213164}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972836980}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606053225}) 2024-08-06T18:19:09.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707786202}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268945122}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1143479371,mdsmap=817224545,mgr=375375032}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=324531614,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=334575961,mon_config_key=2883406912}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4166231994}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3370492230}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=311221516,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1827845780,osdmap=1269523833}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1052720000}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2686201471}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=519985974}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3418669953}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4268964665}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1156535419}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=90} crc {osdmap=4071771349}) 2024-08-06T18:19:09.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/361587101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:09.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: pgmap v1603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:09.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1714871611,config=507150784,health=2077816597,kv=1077530148}) 2024-08-06T18:19:09.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {kv=11,logm=89} crc {kv=1169701999,logm=1272414743}) 2024-08-06T18:19:09.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403702368}) 2024-08-06T18:19:09.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=580189836}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309969450}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371969285}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090047580}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106414157}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095974148}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831979699}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984507092}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357188910}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145306464}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229625650}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186237441}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=806453732}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808529168}) 2024-08-06T18:19:09.937 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3634317104}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=483247400}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568927254}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825396583}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421523470}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970339360}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894802967}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195030136}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869429989}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960963129}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249957449}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557524930}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862529299}) 2024-08-06T18:19:09.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112582149}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=943626556}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3155336851}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=682635022}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156461017}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2755819301}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2854687652}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072541837}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339411591}) 2024-08-06T18:19:09.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010561712}) 2024-08-06T18:19:09.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733980372}) 2024-08-06T18:19:09.940 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108165419}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782450109}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036111200}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965811701}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204950315}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585816328}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446229589}) 2024-08-06T18:19:09.941 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4126774433}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414517977}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893880449}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901020071}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616213164}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972836980}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606053225}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707786202}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268945122}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1143479371,mdsmap=817224545,mgr=375375032}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=324531614,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=334575961,mon_config_key=2883406912}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4166231994}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3370492230}) 2024-08-06T18:19:09.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=311221516,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1827845780,osdmap=1269523833}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1052720000}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2686201471}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=519985974}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3418669953}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4268964665}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1156535419}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=90} crc {osdmap=4071771349}) 2024-08-06T18:19:09.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/361587101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:09.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:10.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: pgmap v1603: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=36,config=2,health=12,kv=50} crc {auth=1714871611,config=507150784,health=2077816597,kv=1077530148}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {kv=11,logm=89} crc {kv=1169701999,logm=1272414743}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=403702368}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=580189836}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2309969450}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1371969285}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2090047580}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4106414157}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2095974148}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=831979699}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=984507092}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3357188910}) 2024-08-06T18:19:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=145306464}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3229625650}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3186237441}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=806453732}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=808529168}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3634317104}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=483247400}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2568927254}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3825396583}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1421523470}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=970339360}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1894802967}) 2024-08-06T18:19:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195030136}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=869429989}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1960963129}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1249957449}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1557524930}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=862529299}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112582149}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=943626556}) 2024-08-06T18:19:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3155336851}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=682635022}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3156461017}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2755819301}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2854687652}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072541837}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=339411591}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1010561712}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733980372}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4108165419}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3782450109}) 2024-08-06T18:19:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1036111200}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2965811701}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1204950315}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=585816328}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2446229589}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4126774433}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1414517977}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1893880449}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3901020071}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3616213164}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=972836980}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1606053225}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1707786202}) 2024-08-06T18:19:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=268945122}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=94,mdsmap=3,mgr=3} crc {logm=1143479371,mdsmap=817224545,mgr=375375032}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mgr=15,mgr_command_descs=1,mgr_metadata=2,mgrstat=10,mon_config_key=72} crc {mgr=324531614,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=334575961,mon_config_key=2883406912}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=4166231994}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3370492230}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=7,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=75} crc {mon_config_key=311221516,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1827845780,osdmap=1269523833}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1052720000}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2686201471}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=519985974}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3418669953}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4268964665}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1156535419}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=90} crc {osdmap=4071771349}) 2024-08-06T18:19:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/361587101' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:10.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:10.277 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:10.278 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:10.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:10.448 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:19:10.449 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:19:10.461 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:10.462 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:10.493 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:10.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:10.631 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:10.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:10.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:10.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:10.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:10.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:10.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.785 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:10.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:11.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:11.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:19:11.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:11.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:11.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:11.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:11.351 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:11.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:11.559 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:11.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:11.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:11.770 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:11 smithi107 ceph-mon[32381]: pgmap v1604: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:11.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:11.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:11 smithi117 ceph-mon[32835]: pgmap v1604: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:11.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:11 smithi086 ceph-mon[29229]: pgmap v1604: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:11.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:12.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:12.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:12.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:12.361 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:12.374 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:12.374 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:12.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:19:12.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:12.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:12.653 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:12.653 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:12.688 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:19:12.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:12.778 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:12.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:12.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:12.975 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:19:12.975 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:19:13.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:13.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:13.176 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:13.176 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:13.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:13.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:13.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:13.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:13.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:19:13.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:13.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:13.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:13 smithi117 ceph-mon[32835]: pgmap v1605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:13 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:13 smithi117 ceph-mon[32835]: osdmap e534: 12 total, 3 up, 11 in 2024-08-06T18:19:13.933 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:13.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:13 smithi107 ceph-mon[32381]: pgmap v1605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:13 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:13 smithi107 ceph-mon[32381]: osdmap e534: 12 total, 3 up, 11 in 2024-08-06T18:19:13.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:13.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:13.967 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:13.967 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:13.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:13 smithi086 ceph-mon[29229]: pgmap v1605: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:13.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:13 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:13.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3083448387' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:13.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:13 smithi086 ceph-mon[29229]: osdmap e534: 12 total, 3 up, 11 in 2024-08-06T18:19:14.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:14.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:14.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:14.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:14.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:14.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:14.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:14.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:14.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:14.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:14.868 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:14.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:15.006 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:15.006 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":534,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:12.686175+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:15.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:15.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:15.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:15.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:15.522 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:19:15.523 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:19:15.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:15.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:15.693 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:15.693 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:15.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:15.825 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:15.825 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:15.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:15 smithi107 ceph-mon[32381]: pgmap v1607: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:15.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/652582791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:15.825 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:15 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:19:15.826 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:15 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:19:15.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:15.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:15.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:15.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:15 smithi086 ceph-mon[29229]: pgmap v1607: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:15.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/652582791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:15.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:15 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:19:15.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:15 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:19:16.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:15 smithi117 ceph-mon[32835]: pgmap v1607: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/652582791' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:15 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:19:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:15 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:19:16.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:16.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:19:16.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:16.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:19:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:16.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:16.539 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:16.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:16.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:16.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:16.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:16.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:17.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:17.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:17.253 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:17.253 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:17.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:17.352 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:17.353 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:17.368 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:17.369 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:17.386 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:17.386 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:17.386 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:17.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:17.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:17.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:17.387 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:17.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:17.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:17.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:17.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:17.621 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:17.621 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:17.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:17.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:17 smithi117 ceph-mon[32835]: pgmap v1608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:17 smithi117 ceph-mon[32835]: osdmap e535: 12 total, 3 up, 11 in 2024-08-06T18:19:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/944130075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:17.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:17.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:17 smithi107 ceph-mon[32381]: pgmap v1608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:17.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:17 smithi107 ceph-mon[32381]: osdmap e535: 12 total, 3 up, 11 in 2024-08-06T18:19:17.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/944130075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:18.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:17 smithi086 ceph-mon[29229]: pgmap v1608: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:18.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:17 smithi086 ceph-mon[29229]: osdmap e535: 12 total, 3 up, 11 in 2024-08-06T18:19:18.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/944130075' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:18.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:18.072 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:18.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:18.158 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:18.202 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:18.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:18.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:18.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:18.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:18.610 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:18.610 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:18.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:18.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:18.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:18.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.877 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.877 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:18.878 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:18.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:19:18.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:19.024 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:19.024 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:19.024 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:19.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:19.181 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:19.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:19.328 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:19.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:19.424 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:19.424 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:19.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:19.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:19.744 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:19 smithi086 ceph-mon[29229]: pgmap v1610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:19.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:19 smithi117 ceph-mon[32835]: pgmap v1610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:19.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:19.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:19 smithi107 ceph-mon[32381]: pgmap v1610: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:20.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:19:20.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:20.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:20.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:20.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:20.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:20.640 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:19:20.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:20.794 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:20.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:20.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1977309246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:20.920 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:20 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:20.920 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:20.920 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1977309246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:20 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:21.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1977309246' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:20 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:21.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:21.175 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:21.180 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:21.181 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:21.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:21.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:21.478 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:21.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:19:21.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:21.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:21.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:21 smithi107 ceph-mon[32381]: pgmap v1611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:21 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:21 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:21 smithi107 ceph-mon[32381]: osdmap e536: 12 total, 3 up, 11 in 2024-08-06T18:19:21.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:21 smithi107 ceph-mon[32381]: osdmap e537: 12 total, 3 up, 11 in 2024-08-06T18:19:21.730 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:19:21.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:21.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:21.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:21 smithi086 ceph-mon[29229]: pgmap v1611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:21 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:21 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:21 smithi086 ceph-mon[29229]: osdmap e536: 12 total, 3 up, 11 in 2024-08-06T18:19:21.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:21 smithi086 ceph-mon[29229]: osdmap e537: 12 total, 3 up, 11 in 2024-08-06T18:19:21.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:22.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:22.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:21 smithi117 ceph-mon[32835]: pgmap v1611: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:22.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:21 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:22.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:21 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:22.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:21 smithi117 ceph-mon[32835]: osdmap e536: 12 total, 3 up, 11 in 2024-08-06T18:19:22.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:21 smithi117 ceph-mon[32835]: osdmap e537: 12 total, 3 up, 11 in 2024-08-06T18:19:22.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:22.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:22.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:22.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:22.519 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:22.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:22.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:22.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:23.028 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:23.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:23.183 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:23.183 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:23.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:23.349 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:23.350 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":537,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:21.621629+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:23.386 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:23.435 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:23.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:23.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:23.730 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:23 smithi117 ceph-mon[32835]: pgmap v1614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:23.731 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2905359066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:23.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:23.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:23 smithi086 ceph-mon[29229]: pgmap v1614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:23.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2905359066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:23.904 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:23 smithi107 ceph-mon[32381]: pgmap v1614: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:23.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2905359066' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:23.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:19:23.970 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:24.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:24.157 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:24.157 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:24.162 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:24.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:24.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:19:24.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:24.384 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:24.385 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:24.417 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:24.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:24.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:24.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:24.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:24.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:24.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:24.448 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:24.448 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:24.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:24.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:24.668 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3980404311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:24.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:24.668 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:24.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:24.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:24.937 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3980404311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:24.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3980404311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:25.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:25.068 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:25.068 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:25.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:25.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:25.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:25.246 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:25.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:25.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:25.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:25.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:25.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:25.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:25.546 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:25.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:25.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:25.743 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:25.743 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:25.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:25 smithi107 ceph-mon[32381]: pgmap v1615: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:25.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.744 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:25 smithi117 ceph-mon[32835]: pgmap v1615: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.882 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:25.882 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:25.882 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:25 smithi086 ceph-mon[29229]: pgmap v1615: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:25.882 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.882 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.882 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:25.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:26.010 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:26.010 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:26.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:26.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:26.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:19:26.250 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:26.381 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:26.408 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:26.408 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:26.408 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:26.408 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:26.464 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:26.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:26.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:26.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:26.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:27.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:27.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:27.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:27.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:19:27.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:27.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:27.460 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:27.501 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:27.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:27.657 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:19:27.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:27.733 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:27.733 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:27.734 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:27 smithi107 ceph-mon[32381]: pgmap v1616: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:27.734 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:19:27.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:27 smithi086 ceph-mon[29229]: pgmap v1616: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:27.804 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:19:27.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:27.849 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:27 smithi117 ceph-mon[32835]: pgmap v1616: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:27.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:19:27.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:28.043 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:28.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:28.110 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:28.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:28.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:28.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:28.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:28.606 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:28.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:19:28.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:28.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:28.853 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:28.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:28.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:28 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:19:28.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:28 smithi086 ceph-mon[29229]: osdmap e538: 12 total, 3 up, 11 in 2024-08-06T18:19:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:28 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:19:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:28 smithi117 ceph-mon[32835]: osdmap e538: 12 total, 3 up, 11 in 2024-08-06T18:19:28.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:28.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:28 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:28.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/830406075' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:19:28.936 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:28 smithi107 ceph-mon[32381]: osdmap e538: 12 total, 3 up, 11 in 2024-08-06T18:19:29.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:29.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:29.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:29.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:29.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:29.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:29.745 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:29 smithi086 ceph-mon[29229]: pgmap v1618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:29.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:29.871 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:29 smithi117 ceph-mon[32835]: pgmap v1618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:29 smithi107 ceph-mon[32381]: pgmap v1618: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:29.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:30.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:30.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:30.313 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:30.380 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:30.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:30.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:30.710 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:30.710 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:30.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:30.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:19:30.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:30.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:19:30.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:31.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:31.130 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:31.130 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":538,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:27.650865+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:20.581196+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:35.361747+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:31.143 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:31.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:31.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:31.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:31.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:31.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:31.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:31.599 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:31.599 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:31.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:31.730 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:31 smithi117 ceph-mon[32835]: pgmap v1619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:31.730 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/108507886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:31.730 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:31 smithi117 ceph-mon[32835]: osdmap e539: 12 total, 3 up, 11 in 2024-08-06T18:19:31.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:31.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:31 smithi086 ceph-mon[29229]: pgmap v1619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:31.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/108507886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:31.851 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:31 smithi086 ceph-mon[29229]: osdmap e539: 12 total, 3 up, 11 in 2024-08-06T18:19:31.949 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:31.949 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:31.949 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:31 smithi107 ceph-mon[32381]: pgmap v1619: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/108507886' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:31 smithi107 ceph-mon[32381]: osdmap e539: 12 total, 3 up, 11 in 2024-08-06T18:19:31.967 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:31.967 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:31.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:31.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:31.986 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:31.988 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:32.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:32.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:32.240 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:32.263 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:32.263 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:32.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:32.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:32.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:32.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:32.618 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:32.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:32.678 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/246706724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:32.678 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:32.679 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:32.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:32.783 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:32.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:32.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:32.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/246706724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:32.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:32.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:32.924 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:32.924 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/246706724' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:33.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:33.156 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:33.157 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:19:33.183 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:33.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:33.405 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:33.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:33.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:33.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:33.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:33.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:33 smithi117 ceph-mon[32835]: pgmap v1621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:33.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:33.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:33.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:33.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:33.927 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:33 smithi086 ceph-mon[29229]: pgmap v1621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:33.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:33 smithi107 ceph-mon[32381]: pgmap v1621: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:33.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:33.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:33.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:33.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:33.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:34.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:34.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:34.290 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:34.290 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:34.291 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:34.291 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:34.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:34.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:34.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:34.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:34.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:34.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:19:34.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:34.805 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:34.805 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:34.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:34.962 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:35.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:35.030 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:35.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:35.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:35.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:35.436 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:35.436 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:35.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:35.525 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:35.526 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:19:35.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:35.674 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:19:35.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:35.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:35 smithi086 ceph-mon[29229]: pgmap v1622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:35.740 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:35.776 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:35.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:35 smithi117 ceph-mon[32835]: pgmap v1622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:35.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:35 smithi107 ceph-mon[32381]: pgmap v1622: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:35.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:36.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:36.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:36.300 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:36.300 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:36.373 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:36.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:36.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:36.740 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:36.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:36 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:36 smithi117 ceph-mon[32835]: osdmap e540: 12 total, 3 up, 11 in 2024-08-06T18:19:36.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:36 smithi117 ceph-mon[32835]: osdmap e541: 12 total, 3 up, 11 in 2024-08-06T18:19:36.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:36.928 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:36 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:36.929 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:36.929 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:36 smithi107 ceph-mon[32381]: osdmap e540: 12 total, 3 up, 11 in 2024-08-06T18:19:36.929 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:36 smithi107 ceph-mon[32381]: osdmap e541: 12 total, 3 up, 11 in 2024-08-06T18:19:36.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:36 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:36.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/449427964' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:36.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:36 smithi086 ceph-mon[29229]: osdmap e540: 12 total, 3 up, 11 in 2024-08-06T18:19:36.979 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:36 smithi086 ceph-mon[29229]: osdmap e541: 12 total, 3 up, 11 in 2024-08-06T18:19:37.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:37.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:37.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:37.369 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:37.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:37.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:37.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:37.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:37 smithi117 ceph-mon[32835]: pgmap v1624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:37.739 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:37 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:19:37.740 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:37 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:19:37.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:37 smithi107 ceph-mon[32381]: pgmap v1624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:37 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:19:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:37 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:19:37.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:37.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:37.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:38.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:38.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:37 smithi086 ceph-mon[29229]: pgmap v1624: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:38.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:37 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:19:38.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:37 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:19:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:38.084 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:38.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:38.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:38.103 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:38.104 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:38.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:19:38.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:38.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:19:38.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:38.343 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:38.361 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:38.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:38.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:38.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:38.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:38.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:19:38.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:19:38.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: osdmap e542: 12 total, 3 up, 11 in 2024-08-06T18:19:38.808 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1706183342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:38.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:38.809 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:38.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:19:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:19:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: osdmap e542: 12 total, 3 up, 11 in 2024-08-06T18:19:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1706183342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: osdmap e542: 12 total, 3 up, 11 in 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1706183342' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.068 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:39.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:39.209 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:39.209 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":542,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:37.678399+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:39.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:39.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:39.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:39.603 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:39.603 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:39.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:39 smithi107 ceph-mon[32381]: pgmap v1627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:39.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3176611460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:39.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:39.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:39.815 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.815 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:39 smithi117 ceph-mon[32835]: pgmap v1627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:39.815 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3176611460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:39.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:39.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:39.850 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:39.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:39.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:39 smithi086 ceph-mon[29229]: pgmap v1627: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:39.850 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3176611460' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:39.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:40.016 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:40.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:40.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:40.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:40.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:40.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:40.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:40.577 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:40.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:19:40.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:40.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:40.821 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:40.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:41.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:41.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:41.050 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:41.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:41.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:41.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:41.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:41.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:41.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:41.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:19:41.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:41.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:41.689 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:41.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:41.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:41 smithi086 ceph-mon[29229]: pgmap v1628: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:41.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:41 smithi086 ceph-mon[29229]: osdmap e543: 12 total, 3 up, 11 in 2024-08-06T18:19:41.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:41 smithi107 ceph-mon[32381]: pgmap v1628: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:41 smithi107 ceph-mon[32381]: osdmap e543: 12 total, 3 up, 11 in 2024-08-06T18:19:41.967 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:41 smithi117 ceph-mon[32835]: pgmap v1628: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:41.967 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:41 smithi117 ceph-mon[32835]: osdmap e543: 12 total, 3 up, 11 in 2024-08-06T18:19:42.024 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:42.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:42.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:42.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:42.379 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:42.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:42.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:42.695 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:19:42.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:42.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:42.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:42.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:42.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:43.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:43.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:43.034 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:43.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:19:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:19:43.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:43.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:43.286 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:43.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:43.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:43.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:43.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:43.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:43.819 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:43 smithi117 ceph-mon[32835]: pgmap v1630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:43.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:43 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:43.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:43.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:43 smithi117 ceph-mon[32835]: osdmap e544: 12 total, 3 up, 11 in 2024-08-06T18:19:43.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:19:43.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:43 smithi107 ceph-mon[32381]: pgmap v1630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:43 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:43 smithi107 ceph-mon[32381]: osdmap e544: 12 total, 3 up, 11 in 2024-08-06T18:19:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:19:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:43.980 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:43.981 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:43.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:43.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:43 smithi086 ceph-mon[29229]: pgmap v1630: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:43.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:43 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:19:43.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2328563595' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:19:43.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:43 smithi086 ceph-mon[29229]: osdmap e544: 12 total, 3 up, 11 in 2024-08-06T18:19:43.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:19:44.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:44.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:44.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:44.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:44.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:44.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:44.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:44.696 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:44.697 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:44.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:44.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1825341127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:44.756 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.757 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:44.822 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:44.847 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1825341127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1825341127' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:44.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:45.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:45.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:45.266 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:45.266 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:45.291 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:45.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:45.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:45.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:45.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:45.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:45 smithi086 ceph-mon[29229]: pgmap v1632: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:45.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:45.866 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:45 smithi117 ceph-mon[32835]: pgmap v1632: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:45.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:45.949 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:45.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:45 smithi107 ceph-mon[32381]: pgmap v1632: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:45.991 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:46.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:46.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:46.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:46.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:46.470 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:46.538 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:46.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:19:46.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:46.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:46.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:46.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:46.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:19:46.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:46.980 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:47.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:47.092 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:47.122 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:47.122 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":545,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:46.628724+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:18:26.468634+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:47.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:47.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:47.291 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:19:47.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:47.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:47.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:47.687 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:47.687 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:47.687 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:47 smithi086 ceph-mon[29229]: pgmap v1633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:47.687 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:47 smithi086 ceph-mon[29229]: osdmap e545: 12 total, 3 up, 11 in 2024-08-06T18:19:47.687 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3767330560' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:47.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:47 smithi117 ceph-mon[32835]: pgmap v1633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:47 smithi117 ceph-mon[32835]: osdmap e545: 12 total, 3 up, 11 in 2024-08-06T18:19:47.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3767330560' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:47.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:47.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:47.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:47 smithi107 ceph-mon[32381]: pgmap v1633: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:47 smithi107 ceph-mon[32381]: osdmap e545: 12 total, 3 up, 11 in 2024-08-06T18:19:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3767330560' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:48.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:48.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:48.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:48.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:48.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:48.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:48.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:48.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:49.007 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:49.007 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:49.089 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:49.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:49.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:49.304 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:49.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:49.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: pgmap v1635: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:19:49.569 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.572 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:49.572 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:49.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: pgmap v1635: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:49.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.572 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:19:49.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:19:49.573 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: pgmap v1635: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:19:49.636 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:19:49.671 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:49.672 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:49.672 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:49.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:49.688 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:49.689 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:49.707 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:49.708 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:49.709 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:49.710 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:49.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:49.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:50.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:50.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:19:50.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:50.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:50.271 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:50.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:50.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:50.530 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/860823071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:50.531 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.531 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/860823071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:50.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.546 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/860823071' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:50.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:50.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:50.925 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:50.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:51.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:51.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:51.153 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:51.153 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:51.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:51.375 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:51.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:51.461 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:51.461 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:51 smithi086 ceph-mon[29229]: pgmap v1636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:51.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:51.526 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:51 smithi117 ceph-mon[32835]: pgmap v1636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:51.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:51.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:51.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:51 smithi107 ceph-mon[32381]: pgmap v1636: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:51.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:51.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:51.759 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:51.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:51.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:52.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:52.080 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:52.080 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:52.080 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:52.080 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:52.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:52.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:52.441 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:52.442 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:19:52.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:52.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:52.641 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:19:52.689 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:19:52.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:52.756 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:52 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:19:52.756 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:52 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:19:52.756 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:52.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:52.903 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:52 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:19:52.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:52 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:19:52.904 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:52.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:19:52.953 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:52 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:19:52.953 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:52 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:19:52.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:19:53.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:53.133 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:53.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:53.258 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:53.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:53.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:53.532 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:53.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:53.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:53.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:53 smithi117 ceph-mon[32835]: pgmap v1637: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:53.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:53 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:53.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:53.900 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:53 smithi117 ceph-mon[32835]: osdmap e546: 12 total, 3 up, 11 in 2024-08-06T18:19:53.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:53.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:53 smithi107 ceph-mon[32381]: pgmap v1637: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:53.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:53 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:53.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:53.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:53 smithi107 ceph-mon[32381]: osdmap e546: 12 total, 3 up, 11 in 2024-08-06T18:19:53.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:53 smithi086 ceph-mon[29229]: pgmap v1637: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:53.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:53 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:19:53.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1472440095' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:19:53.959 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:53 smithi086 ceph-mon[29229]: osdmap e546: 12 total, 3 up, 11 in 2024-08-06T18:19:54.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:54.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:54.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:54.215 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:19:54.229 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:54.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:54.395 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:19:54.396 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":546,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:19:52.638172+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:19:54.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:54.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:54.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:54.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2486812809' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:54.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:54.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:54.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2486812809' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:54.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2486812809' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:19:54.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:19:54.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:19:55.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:55.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:55.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:55.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:55.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:55.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:55.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:55.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:55.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:19:55.808 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:55 smithi117 ceph-mon[32835]: pgmap v1639: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:55.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:55 smithi086 ceph-mon[29229]: pgmap v1639: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:55.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:55.927 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:55.932 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:19:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:19:55.932 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:19:55.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:55.943 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:19:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:19:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:19:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:19:55.944 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:19:55.961 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:19:55.961 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:19:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:19:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:19:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:19:55.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:19:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:55 smithi107 ceph-mon[32381]: pgmap v1639: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:55.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:19:56.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:56.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:56.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:56.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:56.612 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:56.612 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:56.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:56.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:56.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2674133147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:56.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:56 smithi086 ceph-mon[29229]: osdmap e547: 12 total, 3 up, 11 in 2024-08-06T18:19:56.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:19:56.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2674133147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:56.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:56.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:56.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:56 smithi107 ceph-mon[32381]: osdmap e547: 12 total, 3 up, 11 in 2024-08-06T18:19:57.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:57.044 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2674133147' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:19:57.044 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:57.044 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:57.044 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:56 smithi117 ceph-mon[32835]: osdmap e547: 12 total, 3 up, 11 in 2024-08-06T18:19:57.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:57.284 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:19:57.285 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:19:57.286 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:57.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:57.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:57.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:57.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:57.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:57.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:57.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:57 smithi117 ceph-mon[32835]: pgmap v1640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:57 smithi086 ceph-mon[29229]: pgmap v1640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:57.900 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:57.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:19:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:57 smithi107 ceph-mon[32381]: pgmap v1640: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:58.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:58.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:19:58.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:19:58.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:19:58.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:58.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:19:58.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:19:59.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:19:59.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:19:59.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:19:59.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:19:59.459 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:59.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:19:59.614 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:19:59.614 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:19:59.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:19:59.647 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:19:59.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:19:59.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:19:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:19:59 smithi086 ceph-mon[29229]: pgmap v1642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:59.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:19:59.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:19:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:19:59 smithi107 ceph-mon[32381]: pgmap v1642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:19:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:19:59.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:19:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:19:59.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:19:59 smithi117 ceph-mon[32835]: pgmap v1642: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:00.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:00.139 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:00.139 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:19:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:00.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:00.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:00.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:00.669 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:20:00.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:00.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:00 smithi117 ceph-mon[32835]: overall HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:20:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:00.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:00 smithi086 ceph-mon[29229]: overall HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:20:00.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:00.933 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:00.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:00.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:00 smithi107 ceph-mon[32381]: overall HEALTH_WARN 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:20:00.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:01.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:01.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:01.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:01.391 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:01.391 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:01.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:01.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:01.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:01.662 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:01.662 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:01.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:01.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:20:01.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:01.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:01 smithi117 ceph-mon[32835]: pgmap v1643: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:01.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:01 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:01.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:01.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:01 smithi117 ceph-mon[32835]: osdmap e548: 12 total, 3 up, 11 in 2024-08-06T18:20:01.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:01 smithi117 ceph-mon[32835]: osdmap e549: 12 total, 3 up, 11 in 2024-08-06T18:20:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:01 smithi107 ceph-mon[32381]: pgmap v1643: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:01 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:01 smithi107 ceph-mon[32381]: osdmap e548: 12 total, 3 up, 11 in 2024-08-06T18:20:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:01 smithi107 ceph-mon[32381]: osdmap e549: 12 total, 3 up, 11 in 2024-08-06T18:20:02.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:01 smithi086 ceph-mon[29229]: pgmap v1643: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:01 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3141683493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:01 smithi086 ceph-mon[29229]: osdmap e548: 12 total, 3 up, 11 in 2024-08-06T18:20:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:01 smithi086 ceph-mon[29229]: osdmap e549: 12 total, 3 up, 11 in 2024-08-06T18:20:02.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:02.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:02.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:02.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:02.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:02.530 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:02.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:02.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:02.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:02.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:02.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:02.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:02.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:02.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:02.996 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:03.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:03.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:03.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:03.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:03.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:03.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:03.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:03 smithi086 ceph-mon[29229]: pgmap v1646: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:03.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:03.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:03.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:03 smithi107 ceph-mon[32381]: pgmap v1646: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:04.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:04.070 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:03 smithi117 ceph-mon[32835]: pgmap v1646: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:04.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:04.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:04.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:04.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:04.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:04.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:04.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:04.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:04.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:04.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:04.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:20:04.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:05.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:05.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:05.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:05.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:05.541 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:05.541 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:05.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:05.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:05 smithi086 ceph-mon[29229]: pgmap v1647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2112867941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:05.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:05.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:05.855 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:05.855 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:05.855 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:05.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:05.856 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:05.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:05.856 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:05.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:05.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:05.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:05.947 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:05 smithi117 ceph-mon[32835]: pgmap v1647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:05.947 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2112867941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:05.947 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:05.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:05 smithi107 ceph-mon[32381]: pgmap v1647: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2112867941' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:06.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:06.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:06.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:06.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:06.535 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:06.535 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:06.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:06.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:06.798 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:06.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:06.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:07.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:07.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:07.314 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:07.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:07.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:07.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:07.722 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:07.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:07.836 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:07 smithi107 ceph-mon[32381]: pgmap v1648: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:07.908 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:20:07.908 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":549,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:20:01.633903+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:20:07.908 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:07 smithi086 ceph-mon[29229]: pgmap v1648: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:07.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:07.971 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:07 smithi117 ceph-mon[32835]: pgmap v1648: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 326 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:08.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:08.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:08.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:08.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:08.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:20:08.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:08.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:08.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3689569423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:08.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:08.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3689569423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:09.008 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3689569423' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:09.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:09.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:09.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:09.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:09.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:09.812 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:09.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:09 smithi086 ceph-mon[29229]: pgmap v1649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:09.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:09.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:09.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:09 smithi107 ceph-mon[32381]: pgmap v1649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:09.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:20:10.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:09 smithi117 ceph-mon[32835]: pgmap v1649: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:10.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:10.262 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:10.262 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:10.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:10.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:10.489 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:10.524 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:10.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:10.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:10.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:10.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:10.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:10.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:10.914 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:11.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:11.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:11.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:11.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:11.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:11.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:11.696 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:11.697 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:11.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:11.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:11.701 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:20:11.825 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:11.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:11 smithi107 ceph-mon[32381]: pgmap v1650: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:11.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:11 smithi086 ceph-mon[29229]: pgmap v1650: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:11.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:12.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:11 smithi117 ceph-mon[32835]: pgmap v1650: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:12.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:12.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:12.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:12.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:20:12.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:12.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:12 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:12.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:12 smithi117 ceph-mon[32835]: osdmap e550: 12 total, 3 up, 11 in 2024-08-06T18:20:12.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:12.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:12 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:12.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:12.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:12 smithi086 ceph-mon[29229]: osdmap e550: 12 total, 3 up, 11 in 2024-08-06T18:20:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:12 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/503617867' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:12.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:12 smithi107 ceph-mon[32381]: osdmap e550: 12 total, 3 up, 11 in 2024-08-06T18:20:13.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:13.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:13.284 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:13.284 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:13.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:13.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:13.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:13.728 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:13.865 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:13.876 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:13 smithi086 ceph-mon[29229]: pgmap v1652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:13.877 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:13.877 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:13 smithi107 ceph-mon[32381]: pgmap v1652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:13.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:13.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:14.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:14.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:14.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:13 smithi117 ceph-mon[32835]: pgmap v1652: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:14.127 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:14.127 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:14.127 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:14.128 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:14.128 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:14.128 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:14.128 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:14.128 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:14.129 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:14.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:14.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:14.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:20:14.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:14.396 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:14.396 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:14.402 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:14.525 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:14.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:14.710 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:14.710 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:14.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:14.820 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/970781806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:14.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:14.821 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:14.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:14.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/970781806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:14.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:14.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:15.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:15.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/970781806' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:15.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:15.088 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:15.193 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:15.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:20:15.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:15.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:15.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:15.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:15.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:15.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:15.619 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:15.619 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:15.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:15.679 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:15.697 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:15.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:15.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:15.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:15.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:15 smithi086 ceph-mon[29229]: pgmap v1653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:15.904 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:15.905 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:15.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:15 smithi107 ceph-mon[32381]: pgmap v1653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:15.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:15.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:15.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:16.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:16.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:15 smithi117 ceph-mon[32835]: pgmap v1653: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:16.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:16.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:16.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:16.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:16.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:16.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:16.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:17.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:17.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:17.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:17.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:17.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:17.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:17.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:20:17.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:17.639 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:17 smithi117 ceph-mon[32835]: pgmap v1654: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:17.639 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:17 smithi117 ceph-mon[32835]: osdmap e551: 12 total, 3 up, 11 in 2024-08-06T18:20:17.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:17.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:17.926 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:17 smithi107 ceph-mon[32381]: pgmap v1654: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:17.926 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:17 smithi107 ceph-mon[32381]: osdmap e551: 12 total, 3 up, 11 in 2024-08-06T18:20:18.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:18.091 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:18.091 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:18.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:17 smithi086 ceph-mon[29229]: pgmap v1654: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:18.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:17 smithi086 ceph-mon[29229]: osdmap e551: 12 total, 3 up, 11 in 2024-08-06T18:20:18.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:18.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:18.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:18.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:18.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:18.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:18.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:18.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:18.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:19.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:19.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:19.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:19.425 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:20:19.425 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:20:19.426 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:20:19.426 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:20:19.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:19.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:19.518 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:19.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:19.641 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:19.641 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:19.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:19.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:19 smithi086 ceph-mon[29229]: pgmap v1656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:19.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:19.909 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:19 smithi117 ceph-mon[32835]: pgmap v1656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:19 smithi107 ceph-mon[32381]: pgmap v1656: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:20.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:20.151 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:20.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:20.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:20.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:20.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:20.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:20.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:20.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:20.648 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:20:20.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:20.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:20.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:20:20.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:20.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:21.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:20:21.010 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:21.010 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:21.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:20:21.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:21.286 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:20:21.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:21.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:21.288 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":552,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:20:20.643545+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:20:21.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:21.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:21.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:21.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:21.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:21.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:21.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: pgmap v1657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: osdmap e552: 12 total, 3 up, 11 in 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2418585875' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:21 smithi117 ceph-mon[32835]: osdmap e553: 12 total, 3 up, 11 in 2024-08-06T18:20:21.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: pgmap v1657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:21.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:21.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:20:21.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: osdmap e552: 12 total, 3 up, 11 in 2024-08-06T18:20:21.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2418585875' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:21.897 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:21 smithi107 ceph-mon[32381]: osdmap e553: 12 total, 3 up, 11 in 2024-08-06T18:20:22.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:22.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: pgmap v1657: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:22.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:22.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/262461157' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:20:22.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: osdmap e552: 12 total, 3 up, 11 in 2024-08-06T18:20:22.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2418585875' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:22.014 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:21 smithi086 ceph-mon[29229]: osdmap e553: 12 total, 3 up, 11 in 2024-08-06T18:20:22.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:22.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:22.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:22.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:22.644 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:22.644 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:22.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:22.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:20:22.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:22.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:22.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:22.974 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:22.974 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:23.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:23.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:23.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:23.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:23.342 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:23.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:23.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:23.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:20:23.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:23.748 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:23 smithi107 ceph-mon[32381]: pgmap v1660: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:23.748 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:23 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:20:23.748 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:23 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:20:23.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:23.749 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:23.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:23 smithi117 ceph-mon[32835]: pgmap v1660: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:23 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:20:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:23 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:20:23.902 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:23.942 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:23.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:24.033 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:24.033 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:24.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:23 smithi086 ceph-mon[29229]: pgmap v1660: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:24.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:23 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:20:24.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:23 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:20:24.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:24.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:24.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:24.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:24.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:24.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:24.946 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:25.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:25.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:25.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:25.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:25.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:20:25.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:25.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:25.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:25.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:25.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:25.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:25.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:25.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:25.771 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:25 smithi107 ceph-mon[32381]: pgmap v1661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:25.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:26.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:26.042 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:25 smithi117 ceph-mon[32835]: pgmap v1661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:26.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:25 smithi086 ceph-mon[29229]: pgmap v1661: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:26.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:26.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:26.378 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:26.378 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:26.379 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:26.395 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:26.396 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:26.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:26.422 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:26.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:26.424 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:26.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:26.575 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:26.575 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:26.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:26.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:26 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:20:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:26 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:20:26.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1504966322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:26 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:20:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:26 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:20:26.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1504966322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:26.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:26 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:20:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:26 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:20:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1504966322' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:27.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:27.159 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:27.159 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:27.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:27.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:27.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:27.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:27.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:27.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:27.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:27.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:27.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:27.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:27.671 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:20:27.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:27.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:27.904 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:27.904 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:27.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:27 smithi086 ceph-mon[29229]: pgmap v1662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:27.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.905 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:27.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:27 smithi107 ceph-mon[32381]: pgmap v1662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:27.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:28.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:28.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:27 smithi117 ceph-mon[32835]: pgmap v1662: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:28.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:28.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:28.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:28.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:28.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:28.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:28.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:28.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:28.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:28.713 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:28 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:20:28.714 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:28.714 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:28 smithi117 ceph-mon[32835]: osdmap e554: 12 total, 3 up, 11 in 2024-08-06T18:20:28.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:28.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:28 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:20:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:28.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:28 smithi107 ceph-mon[32381]: osdmap e554: 12 total, 3 up, 11 in 2024-08-06T18:20:29.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:29.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:28 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:20:29.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4144116839' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:29.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:28 smithi086 ceph-mon[29229]: osdmap e554: 12 total, 3 up, 11 in 2024-08-06T18:20:29.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:20:29.230 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:29.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:29.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:29.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:29.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:29.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:29.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:29.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:29.674 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:29.674 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:29.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:29 smithi107 ceph-mon[32381]: pgmap v1664: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:29.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:29.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:29.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:29.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:29.925 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:29 smithi086 ceph-mon[29229]: pgmap v1664: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:30.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:29 smithi117 ceph-mon[32835]: pgmap v1664: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:30.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:30.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:30.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:30.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:30.367 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:20:30.367 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":554,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:20:27.668998+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:20:30.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:30.519 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:20:30.519 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:20:30.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:30.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:30.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:30.834 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1940885882' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:30.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1940885882' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1940885882' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:30.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:31.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:31.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:31.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:20:31.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:31.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:31.496 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:31.496 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:31.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:31.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:31 smithi086 ceph-mon[29229]: pgmap v1665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:31 smithi086 ceph-mon[29229]: osdmap e555: 12 total, 3 up, 11 in 2024-08-06T18:20:31.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:31.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:31.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:31.872 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:31 smithi117 ceph-mon[32835]: pgmap v1665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:31 smithi117 ceph-mon[32835]: osdmap e555: 12 total, 3 up, 11 in 2024-08-06T18:20:31.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:20:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:31 smithi107 ceph-mon[32381]: pgmap v1665: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:31 smithi107 ceph-mon[32381]: osdmap e555: 12 total, 3 up, 11 in 2024-08-06T18:20:32.000 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:32.077 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:32.113 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:32.135 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:32.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:32.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:32.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:32.600 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:32.600 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:32.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:32.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:32.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:32.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:32.921 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:33.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:33.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:33.235 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:33.235 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:33.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:33.452 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:33.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:33.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:33.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:33.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:33 smithi086 ceph-mon[29229]: pgmap v1667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:33.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:33.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:33.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:33 smithi117 ceph-mon[32835]: pgmap v1667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:33 smithi107 ceph-mon[32381]: pgmap v1667: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:33.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:34.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:34.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:34.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:34.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:34.534 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:34.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:34.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:34.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:34.936 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:34.936 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:34.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:34.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:35.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:20:35.046 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:35.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:35.208 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:35.209 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:35.209 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:35.227 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:35.227 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:35.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:35.227 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:35.228 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:35.246 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:35.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:35.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:35.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:35.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:35.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:35.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:35.609 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:423,from:6,reservation_nonce:17)", 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 556.38375775500003, 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014934700000000001, 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.610 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018877+0000", 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.611 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018879+0000", 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1619999999999998e-06 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018888+0000", 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1570000000000006e-06 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018906+0000", 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7773000000000001e-05 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.612 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018917+0000", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1262e-05 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018984+0000", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6549000000000003e-05 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.018999+0000", 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5593000000000001e-05 2024-08-06T18:20:35.613 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019026+0000", 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6851000000000001e-05 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:423/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 556.38292858199998, 2024-08-06T18:20:35.614 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014629899999999999, 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.615 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019706+0000", 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019707+0000", 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6499999999999999e-07 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.616 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019740+0000", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3853999999999998e-05 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019750+0000", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019761+0000", 2024-08-06T18:20:35.617 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1636e-05 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019809+0000", 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7369e-05 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019820+0000", 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1606999999999999e-05 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.618 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.019852+0000", 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2240000000000003e-05 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:423,from:6,reservation_nonce:0)", 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 556.381972787, 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000163024, 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.619 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.620 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020662+0000", 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.86e-07 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020664+0000", 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.579e-06 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020685+0000", 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1350999999999999e-05 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.621 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020689+0000", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2910000000000001e-06 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020805+0000", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00011613 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020816+0000", 2024-08-06T18:20:35.622 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.048e-05 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:11:19.020825+0000", 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0070000000000003e-06 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.623 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:441,from:2,reservation_nonce:16)", 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 498.78271787699998, 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011107860000000001, 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619917+0000", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619918+0000", 2024-08-06T18:20:35.625 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619927+0000", 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619942+0000", 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5214999999999999e-05 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.626 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.619950+0000", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9519999999999994e-06 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620000+0000", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9979999999999999e-05 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620004+0000", 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0589999999999996e-06 2024-08-06T18:20:35.627 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620978+0000", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00097387000000000005 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.620993+0000", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4676e-05 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.621028+0000", 2024-08-06T18:20:35.628 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4950999999999998e-05 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:441,from:2,reservation_nonce:0)", 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 498.78047426099999, 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111129, 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.629 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622160+0000", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622161+0000", 2024-08-06T18:20:35.630 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7599999999999998e-07 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622163+0000", 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.951e-06 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622167+0000", 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8650000000000003e-06 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.631 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622170+0000", 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5260000000000002e-06 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622240+0000", 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9622000000000005e-05 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622257+0000", 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7754e-05 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.632 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:12:16.622272+0000", 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4134999999999999e-05 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:457,from:2,reservation_nonce:17)", 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.049798313, 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010675699999999999, 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.633 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352836+0000", 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.634 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352837+0000", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.7300000000000005e-07 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352842+0000", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2769999999999999e-06 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352852+0000", 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:20:35.635 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352859+0000", 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6959999999999999e-06 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352906+0000", 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7364999999999998e-05 2024-08-06T18:20:35.636 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352919+0000", 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3159999999999999e-05 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.352943+0000", 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3828e-05 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.637 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:457/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.04915851700002, 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3411000000000004e-05, 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.638 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353476+0000", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353477+0000", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4199999999999995e-07 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353481+0000", 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.5820000000000003e-06 2024-08-06T18:20:35.639 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353493+0000", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1887e-05 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353499+0000", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6010000000000002e-06 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353543+0000", 2024-08-06T18:20:35.640 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3927000000000001e-05 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353549+0000", 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5549999999999999e-06 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353570+0000", 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0217e-05 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.641 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:457,from:2,reservation_nonce:0)", 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 417.04843818199998, 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012299899999999999, 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.642 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354197+0000", 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.39e-07 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.643 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354199+0000", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5150000000000001e-06 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354206+0000", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0090000000000001e-06 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354219+0000", 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2714999999999999e-05 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.644 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354300+0000", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1031000000000004e-05 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354310+0000", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0359e-05 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.354320+0000", 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0310000000000004e-06 2024-08-06T18:20:35.645 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:476,from:2,reservation_nonce:18)", 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.981577351, 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012406, 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.646 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.649 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.649 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421057+0000", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421059+0000", 2024-08-06T18:20:35.650 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.471e-06 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421067+0000", 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0600000000000008e-06 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421082+0000", 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5305e-05 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.651 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421093+0000", 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0558e-05 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421142+0000", 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8767999999999998e-05 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421156+0000", 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4027e-05 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.652 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421181+0000", 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5871e-05 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:476/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.980976199, 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000119988, 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.653 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.654 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421659+0000", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4099999999999999e-07 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421663+0000", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3649999999999997e-06 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421673+0000", 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0149000000000001e-05 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.655 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421681+0000", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9920000000000007e-06 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421726+0000", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4246999999999998e-05 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421737+0000", 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1796e-05 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.656 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421778+0000", 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0998000000000003e-05 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:476,from:2,reservation_nonce:0)", 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 350.98031278100001, 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111465, 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.657 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.658 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422322+0000", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2300000000000002e-07 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422325+0000", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4109999999999999e-06 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422331+0000", 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2530000000000001e-06 2024-08-06T18:20:35.659 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422336+0000", 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.558e-06 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422406+0000", 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.9059000000000004e-05 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.660 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422420+0000", 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422433+0000", 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3499000000000001e-05 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:509,from:2,reservation_nonce:20)", 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:20:35.661 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 198.241640257, 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011134249999999999, 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.662 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160996+0000", 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161004+0000", 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2330000000000006e-06 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.663 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161020+0000", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5559000000000001e-05 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161028+0000", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2819999999999996e-06 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161088+0000", 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9743e-05 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.664 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161092+0000", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7160000000000002e-06 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162061+0000", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00096807299999999996 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162076+0000", 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5928999999999999e-05 2024-08-06T18:20:35.665 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162108+0000", 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1380000000000001e-05 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:509/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 198.24005498700001, 2024-08-06T18:20:35.666 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112194, 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.667 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8400000000000004e-07 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162592+0000", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1977000000000001e-05 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162605+0000", 2024-08-06T18:20:35.668 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2602000000000001e-05 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162614+0000", 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.918e-06 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162663+0000", 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9168000000000001e-05 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.669 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162670+0000", 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.396e-06 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162692+0000", 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1449000000000001e-05 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:509,from:2,reservation_nonce:0)", 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:20:35.670 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 198.23941111900001, 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112296, 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.671 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9499999999999998e-07 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163226+0000", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1940000000000001e-06 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163231+0000", 2024-08-06T18:20:35.672 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1819999999999997e-06 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163235+0000", 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8290000000000001e-06 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163305+0000", 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0358000000000006e-05 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.673 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163322+0000", 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6384000000000001e-05 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163336+0000", 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4054e-05 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:553,from:2,reservation_nonce:22)", 2024-08-06T18:20:35.674 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 13.015026936, 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012249200000000001, 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.675 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387609+0000", 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.48e-06 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387617+0000", 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9510000000000006e-06 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.676 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387631+0000", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4141000000000001e-05 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387640+0000", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3799999999999994e-06 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387689+0000", 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9407000000000003e-05 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.677 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387705+0000", 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5583e-05 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387730+0000", 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5550000000000001e-05 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.678 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:553/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 13.014467384, 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012141, 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.679 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388168+0000", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9500000000000002e-07 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388172+0000", 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2220000000000004e-06 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.680 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388181+0000", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0610000000000002e-06 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388188+0000", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.153e-06 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388247+0000", 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8544000000000001e-05 2024-08-06T18:20:35.681 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388258+0000", 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0879999999999999e-05 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388289+0000", 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0954999999999998e-05 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.682 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:553,from:2,reservation_nonce:0)", 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 13.013788681999999, 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110738, 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.683 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4999999999999999e-07 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.684 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388848+0000", 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7349999999999999e-06 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388854+0000", 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5450000000000003e-06 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388859+0000", 2024-08-06T18:20:35.685 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0520000000000001e-06 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388937+0000", 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8552000000000003e-05 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388949+0000", 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1409e-05 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.686 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388957+0000", 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1950000000000003e-06 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:553,from:6,reservation_nonce:24)", 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.9983942149999994, 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014068199999999999, 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.687 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.688 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404242+0000", 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5540000000000001e-06 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404251+0000", 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2010000000000005e-06 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404270+0000", 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9028000000000001e-05 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.689 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404279+0000", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5220000000000007e-06 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404340+0000", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1649000000000006e-05 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404355+0000", 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4484999999999999e-05 2024-08-06T18:20:35.690 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404381+0000", 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6242999999999999e-05 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:553/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.9976514069999993, 2024-08-06T18:20:35.691 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101147, 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.692 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404984+0000", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.58e-07 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404995+0000", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.095e-05 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405004+0000", 2024-08-06T18:20:35.693 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3200000000000006e-06 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405009+0000", 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0930000000000002e-06 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405053+0000", 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3495000000000003e-05 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.694 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405060+0000", 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4619999999999999e-06 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405084+0000", 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4369e-05 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.695 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:553,from:6,reservation_nonce:0)", 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.405809+0000", 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 9.9968254460000008, 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0502999999999997e-05, 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405809+0000", 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.696 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405809+0000", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405810+0000", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6899999999999998e-07 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405812+0000", 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5600000000000001e-06 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.697 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405818+0000", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.122e-06 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405823+0000", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.809e-06 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405883+0000", 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9919e-05 2024-08-06T18:20:35.698 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405891+0000", 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2390000000000002e-06 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405900+0000", 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4849999999999992e-06 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:20:35.699 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:20:35.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:35.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:35 smithi086 ceph-mon[29229]: pgmap v1668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1501668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:35 smithi117 ceph-mon[32835]: pgmap v1668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1501668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:35.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:35 smithi107 ceph-mon[32381]: pgmap v1668: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1501668113' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:36.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:36.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:36.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:36.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:36.512 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:36.512 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:36.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:36.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:36.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:36.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:20:36.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:36.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:36.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:36.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:36.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:36.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:36.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:37.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:37.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:37.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:37.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:37.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:37.601 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:37.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:37.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:37.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:37 smithi086 ceph-mon[29229]: pgmap v1669: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:37.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:37.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:37.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:37.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:37 smithi117 ceph-mon[32835]: pgmap v1669: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:37 smithi107 ceph-mon[32381]: pgmap v1669: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:38.017 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:38.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:38.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:38.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:38.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:38.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:38.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:38.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:38.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:38.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:20:38.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:38.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:38.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:39.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:39.138 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:39.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:39.288 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:39.288 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:39.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:39.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:39.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:39.697 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:39.697 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:39.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:39.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:39 smithi117 ceph-mon[32835]: pgmap v1670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:39.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:39 smithi107 ceph-mon[32381]: pgmap v1670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:39.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:39.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:40.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:39 smithi086 ceph-mon[29229]: pgmap v1670: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:40.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:40.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:40.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:40.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:40.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:40.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:40.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:40.702 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:20:40.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:40.789 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:40.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:40.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:41.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:41.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:20:41.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:41.191 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:41.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:41.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:41.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:41.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:41.726 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:41 smithi086 ceph-mon[29229]: pgmap v1671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:41.727 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:41.727 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:41.727 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:41 smithi086 ceph-mon[29229]: osdmap e556: 12 total, 3 up, 11 in 2024-08-06T18:20:41.727 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:41 smithi086 ceph-mon[29229]: osdmap e557: 12 total, 3 up, 11 in 2024-08-06T18:20:41.782 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:41.835 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:41 smithi107 ceph-mon[32381]: pgmap v1671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:41.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:41 smithi107 ceph-mon[32381]: osdmap e556: 12 total, 3 up, 11 in 2024-08-06T18:20:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:41 smithi107 ceph-mon[32381]: osdmap e557: 12 total, 3 up, 11 in 2024-08-06T18:20:41.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:41.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:41 smithi117 ceph-mon[32835]: pgmap v1671: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:41.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:41.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/446367311' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:20:41.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:41 smithi117 ceph-mon[32835]: osdmap e556: 12 total, 3 up, 11 in 2024-08-06T18:20:41.990 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:41 smithi117 ceph-mon[32835]: osdmap e557: 12 total, 3 up, 11 in 2024-08-06T18:20:42.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:42.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:42.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:42.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:42.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:42.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:42.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:42.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:42.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:42.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:42.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:42.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:42.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:42.918 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:42.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:42.953 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:20:42.953 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":557,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:20:41.645306+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:20:43.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:43.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:43.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:43.434 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:43.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:43.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:43 smithi107 ceph-mon[32381]: pgmap v1674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:43.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2943665463' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:43.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:43.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:44.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:43 smithi117 ceph-mon[32835]: pgmap v1674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:44.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2943665463' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:44.063 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:43 smithi086 ceph-mon[29229]: pgmap v1674: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:44.064 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2943665463' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:44.099 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:44.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:20:44.112 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:44.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:44.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:20:44.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:44.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:44.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:44.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:44.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:44.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:44.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:44.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:44.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:44.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:44.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:45.027 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:45.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:45.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:45.199 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:45.200 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:45.200 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:45.200 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:45.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:45.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:45.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:45.722 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:45.796 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:45 smithi086 ceph-mon[29229]: pgmap v1675: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:45.796 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:45.796 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:45.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:45 smithi107 ceph-mon[32381]: pgmap v1675: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:45.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:46.014 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:45 smithi117 ceph-mon[32835]: pgmap v1675: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:46.093 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:46.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:46.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:46.348 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:46.348 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:46.348 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:46.349 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:46.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:46.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:46.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:46.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:46.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:20:46.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:46.818 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3922086016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:46.818 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:46.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:46.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3922086016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:46.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:47.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:47.061 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3922086016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:47.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:47.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:47.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:47.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:47.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:47.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:47.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:47.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:47.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:47.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:47.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:47 smithi086 ceph-mon[29229]: pgmap v1676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:47.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:47.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:47.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:48.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:47 smithi117 ceph-mon[32835]: pgmap v1676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:48.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:47 smithi107 ceph-mon[32381]: pgmap v1676: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:48.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:48.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:48.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:48.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:48.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:48.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:48.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:48.808 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:20:48.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:48.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:20:49.002 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:49.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:49.092 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:20:49.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:20:49.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:49.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:49.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:49.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:49.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:49.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:49.859 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:49 smithi107 ceph-mon[32381]: pgmap v1677: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:49.859 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:49.859 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:49.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:49 smithi086 ceph-mon[29229]: pgmap v1677: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:49.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:49.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:49.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:50.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:49 smithi117 ceph-mon[32835]: pgmap v1677: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:50.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:50.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:50.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:50.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:50.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:50.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:20:50.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:20:50.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:50.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:50.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:50.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:50.880 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:50.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:20:50.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:50.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:20:50.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:20:50.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:50.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:50.927 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:20:51.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:51.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:20:51.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:20:51.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:20:51.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:20:51.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:51.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:51.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:51.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:51.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:51.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:51.571 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:51.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:51.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:51 smithi086 ceph-mon[29229]: pgmap v1678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:51 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:51 smithi086 ceph-mon[29229]: osdmap e558: 12 total, 3 up, 11 in 2024-08-06T18:20:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:51 smithi086 ceph-mon[29229]: osdmap e559: 12 total, 3 up, 11 in 2024-08-06T18:20:51.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:52.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:52.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:51 smithi117 ceph-mon[32835]: pgmap v1678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:52.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:51 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:52.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:52.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:51 smithi117 ceph-mon[32835]: osdmap e558: 12 total, 3 up, 11 in 2024-08-06T18:20:52.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:51 smithi117 ceph-mon[32835]: osdmap e559: 12 total, 3 up, 11 in 2024-08-06T18:20:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:51 smithi107 ceph-mon[32381]: pgmap v1678: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:51 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:20:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3754467406' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:20:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:51 smithi107 ceph-mon[32381]: osdmap e558: 12 total, 3 up, 11 in 2024-08-06T18:20:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:51 smithi107 ceph-mon[32381]: osdmap e559: 12 total, 3 up, 11 in 2024-08-06T18:20:52.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:52.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:52.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:52.523 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:52.523 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:52.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:52.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:52.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:52.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:52.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:52.935 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:20:52.935 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":559,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:20:51.648087+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:36.987874+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:37.142586+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:20:53.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:53.030 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:53.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:53.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:53.173 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:53.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:53.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:53.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:53.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:53.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:53 smithi117 ceph-mon[32835]: pgmap v1681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1642407274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:53.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:53.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:20:54.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:53 smithi086 ceph-mon[29229]: pgmap v1681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:54.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1642407274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:54.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:53 smithi107 ceph-mon[32381]: pgmap v1681: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1642407274' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:20:54.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:54.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:54.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:54.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:54.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:54.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:54.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:54.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:54 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:20:54.839 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:54 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:20:54.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:54.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:54.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:54.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:54.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:55.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:54 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:20:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:54 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:20:55.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:55.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:54 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:20:55.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:54 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:20:55.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:55.412 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:55.416 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:55.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:55.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:55.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:55.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:55.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:55.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:55 smithi086 ceph-mon[29229]: pgmap v1682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:55.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:55 smithi086 ceph-mon[29229]: osdmap e560: 12 total, 3 up, 11 in 2024-08-06T18:20:55.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:55 smithi107 ceph-mon[32381]: pgmap v1682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:55 smithi107 ceph-mon[32381]: osdmap e560: 12 total, 3 up, 11 in 2024-08-06T18:20:55.988 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:55 smithi117 ceph-mon[32835]: pgmap v1682: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:55.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:55 smithi117 ceph-mon[32835]: osdmap e560: 12 total, 3 up, 11 in 2024-08-06T18:20:56.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:56.141 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:20:56.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:56.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:56.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:56.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:56.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:56.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:56.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:56.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:56.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:56.826 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:20:56.827 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:20:56.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:20:56.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:57.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:57.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:57.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:57.179 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:57.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:57.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:57.329 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:57.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:57.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:57.601 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:57.601 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:57.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:57.739 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:20:57.740 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:20:57.740 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:20:57.759 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:20:57.760 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:20:57.760 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:20:57.760 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:20:57.760 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:20:57.779 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:20:57.779 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:20:57.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:20:57.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:20:57.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:20:57.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:57.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:20:57.781 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:20:57.781 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:20:57.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:57.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:57 smithi086 ceph-mon[29229]: pgmap v1684: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:57.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:57 smithi086 ceph-mon[29229]: osdmap e561: 12 total, 3 up, 11 in 2024-08-06T18:20:57.863 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:57.864 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:57.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:20:57.864 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:20:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:57 smithi117 ceph-mon[32835]: pgmap v1684: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:57 smithi117 ceph-mon[32835]: osdmap e561: 12 total, 3 up, 11 in 2024-08-06T18:20:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:57 smithi107 ceph-mon[32381]: pgmap v1684: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:57 smithi107 ceph-mon[32381]: osdmap e561: 12 total, 3 up, 11 in 2024-08-06T18:20:57.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:20:58.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:58.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:58.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:20:58.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:20:58.549 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:20:58.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:20:58.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:20:58.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3823592380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:58.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.806 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.807 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3823592380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:58.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.856 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:20:58.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:20:58.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:20:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:20:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3823592380' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:20:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:58.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:20:59.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:20:59.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:20:59.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:20:59.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:20:59.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:20:59.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:20:59.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:20:59.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:20:59.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:20:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:20:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:59 smithi086 ceph-mon[29229]: pgmap v1686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:59 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:20:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:59 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:20:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:20:59 smithi086 ceph-mon[29229]: osdmap e562: 12 total, 3 up, 11 in 2024-08-06T18:20:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:59 smithi117 ceph-mon[32835]: pgmap v1686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:59 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:20:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:59 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:20:59.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:20:59 smithi117 ceph-mon[32835]: osdmap e562: 12 total, 3 up, 11 in 2024-08-06T18:20:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:59 smithi107 ceph-mon[32381]: pgmap v1686: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:20:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:59 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:20:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:59 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:20:59.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:20:59 smithi107 ceph-mon[32381]: osdmap e562: 12 total, 3 up, 11 in 2024-08-06T18:20:59.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:00.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:00.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:00.297 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:00.298 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:20:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:00.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:00.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:00.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:21:00.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:00.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:00.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:01.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:01.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:01.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:01.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:01.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:01.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:21:01.762 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:21:01.762 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:21:01.762 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:21:01.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:01.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:01 smithi086 ceph-mon[29229]: pgmap v1688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:01.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:01.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:01 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:01.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:01.902 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:01 smithi086 ceph-mon[29229]: osdmap e563: 12 total, 3 up, 11 in 2024-08-06T18:21:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:01 smithi107 ceph-mon[32381]: pgmap v1688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:01 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:01 smithi107 ceph-mon[32381]: osdmap e563: 12 total, 3 up, 11 in 2024-08-06T18:21:02.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:01 smithi117 ceph-mon[32835]: pgmap v1688: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:01 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1529007148' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:01 smithi117 ceph-mon[32835]: osdmap e563: 12 total, 3 up, 11 in 2024-08-06T18:21:02.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:02.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:02.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:02.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:02.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:02.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:02.412 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:02.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:21:02.520 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:21:02.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:02.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:02.812 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:02.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:03.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:03.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:03.352 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:03.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:03.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:03.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:03.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:03 smithi086 ceph-mon[29229]: pgmap v1690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:03.820 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:03.820 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:03.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:03.917 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:03 smithi117 ceph-mon[32835]: pgmap v1690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:03 smithi107 ceph-mon[32381]: pgmap v1690: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:04.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:04.027 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:04.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:04.201 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:04.202 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":563,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:01.650993+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:04.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:04.430 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:04.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:04.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:04.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:04.857 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4149421172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:04.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4149421172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:04.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4149421172' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:04.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:05.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:05.171 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:05.171 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:05.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:05.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:05.401 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:05.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:05.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:05.514 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:05.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:21:05.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:05.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:05.780 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:05 smithi086 ceph-mon[29229]: pgmap v1691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:05.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:05.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:05.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:05.923 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:05 smithi107 ceph-mon[32381]: pgmap v1691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:06.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:06.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:05 smithi117 ceph-mon[32835]: pgmap v1691: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:06.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:06.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:06.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:06.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:06.589 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:06.589 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:06.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:06.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:06.894 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:06.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:07.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:07.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:07.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:07.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:07.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:07.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:07.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:07.738 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:21:07.739 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:21:07.739 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:21:07.739 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:21:07.739 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:21:07.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:07.795 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:07 smithi117 ceph-mon[32835]: pgmap v1692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:07.826 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:07 smithi086 ceph-mon[29229]: pgmap v1692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:07.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:07.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:07 smithi107 ceph-mon[32381]: pgmap v1692: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:08.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:08.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:08.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:08.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:08.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:08.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:08.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:08.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:21:08.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:08.817 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:08.818 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:08.818 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:08.836 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:08.836 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:08.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:08.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:08.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:08.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:08.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:08.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:08.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:08.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:08.857 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:08.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:08.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:08.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:09.067 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:09.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:09.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:09.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:09.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:09.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:09.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:09.519 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:09.519 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:09.647 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:09.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:09.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:09 smithi086 ceph-mon[29229]: pgmap v1693: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:09.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3852721815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:09.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:09.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:09.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:09.939 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:09 smithi117 ceph-mon[32835]: pgmap v1693: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:09.939 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3852721815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:09.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:09.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:09 smithi107 ceph-mon[32381]: pgmap v1693: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3852721815' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:10.048 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:10.127 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:10.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:10.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:10.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:10.472 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:10.473 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:10.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:10.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:10.739 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:10.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:10.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:10.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:10.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:10.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:11.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:11.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:11.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:11.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:11.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:11.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:11.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:11.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:21:11.539 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:11.539 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:11.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:11.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:11.807 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:11 smithi107 ceph-mon[32381]: pgmap v1694: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:11.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:11 smithi086 ceph-mon[29229]: pgmap v1694: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:11.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:12.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:11 smithi117 ceph-mon[32835]: pgmap v1694: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:12.184 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:12.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:12.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:12.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:12.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:12.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:12.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:12.727 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:12.727 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":563,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:01.650993+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:19:51.717684+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:12.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:12.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:13.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:13.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:13.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:13.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:13.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:13.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:13.833 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:13 smithi107 ceph-mon[32381]: pgmap v1695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:13.833 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4156628576' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:13.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:14.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:13 smithi086 ceph-mon[29229]: pgmap v1695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:14.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4156628576' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:14.078 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:13 smithi117 ceph-mon[32835]: pgmap v1695: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4156628576' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:14.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:14.338 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:14.338 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:14.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:14.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:14.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:14.715 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:14.749 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:14.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:14.966 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:14.966 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:15.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:15.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:15.220 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:21:15.220 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:21:15.220 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:15.220 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:15.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:15.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:15.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:15.560 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:15.560 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:15.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:15.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:15.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:15.704 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:21:15.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:15.822 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:15 smithi086 ceph-mon[29229]: pgmap v1696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:15.823 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:15.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:15.823 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:15.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:15.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:15 smithi107 ceph-mon[32381]: pgmap v1696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:15.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:16.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:15 smithi117 ceph-mon[32835]: pgmap v1696: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:16.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:16.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:16.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:16.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:16.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:16.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:16.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:16.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:16.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:16.760 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:16 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:16.760 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:16.761 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:16 smithi107 ceph-mon[32381]: osdmap e564: 12 total, 3 up, 11 in 2024-08-06T18:21:16.761 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:16 smithi107 ceph-mon[32381]: osdmap e565: 12 total, 3 up, 11 in 2024-08-06T18:21:16.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:16.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:16 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:16.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:16.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:16 smithi086 ceph-mon[29229]: osdmap e564: 12 total, 3 up, 11 in 2024-08-06T18:21:16.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:16 smithi086 ceph-mon[29229]: osdmap e565: 12 total, 3 up, 11 in 2024-08-06T18:21:17.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:16 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2576560230' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:16 smithi117 ceph-mon[32835]: osdmap e564: 12 total, 3 up, 11 in 2024-08-06T18:21:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:16 smithi117 ceph-mon[32835]: osdmap e565: 12 total, 3 up, 11 in 2024-08-06T18:21:17.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:17.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:17.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:17.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:17.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:17.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:21:17.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:17.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:17.649 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:17.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:17.706 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:17.706 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:17.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:17.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:17 smithi117 ceph-mon[32835]: pgmap v1698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:17 smithi107 ceph-mon[32381]: pgmap v1698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:17.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:17.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:17 smithi086 ceph-mon[29229]: pgmap v1698: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:17.986 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:17.986 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:18.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:18.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:18.315 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:18.315 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:18.315 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:18.324 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:18.325 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:18.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:18.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:18.325 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:18.344 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:18.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:18.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:18.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:18.589 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:18.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:18.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:18.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/555574374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:18.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:18.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:18.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:18.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:18.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:19.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/555574374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:19.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:19.030 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:19.030 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/555574374' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:19.278 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:19.278 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:19.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:19.358 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:19.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:19.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:19.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:19.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:19.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:21:19.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:19.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:19 smithi086 ceph-mon[29229]: pgmap v1700: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:19.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:19 smithi107 ceph-mon[32381]: pgmap v1700: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:19.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:19 smithi117 ceph-mon[32835]: pgmap v1700: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:19.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:19.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:21:20.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:20.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:20.265 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:20.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:20.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:20.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:20.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:20.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:20.883 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:21.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:21.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:21:21.084 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:21.084 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":566,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:20.716104+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:21.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:21.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:21.326 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:21.326 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:21.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:21.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:21.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:21.719 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:21.719 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:21.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:21.814 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: pgmap v1701: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: osdmap e566: 12 total, 3 up, 11 in 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/618158439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:21 smithi117 ceph-mon[32835]: osdmap e567: 12 total, 3 up, 11 in 2024-08-06T18:21:21.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: pgmap v1701: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: osdmap e566: 12 total, 3 up, 11 in 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/618158439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:21 smithi107 ceph-mon[32381]: osdmap e567: 12 total, 3 up, 11 in 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: pgmap v1701: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: osdmap e566: 12 total, 3 up, 11 in 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/618158439' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:22.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:21 smithi086 ceph-mon[29229]: osdmap e567: 12 total, 3 up, 11 in 2024-08-06T18:21:22.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:22.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:22.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:22.451 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:22.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:21:22.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:22.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:22.632 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:22.704 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:22.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:22.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:23.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:23.156 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:23.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:23.419 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:23.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:23.577 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:23.577 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:23.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:23.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:23.840 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:23 smithi086 ceph-mon[29229]: pgmap v1704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:23.856 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:23 smithi107 ceph-mon[32381]: pgmap v1704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:23.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:23.976 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:23 smithi117 ceph-mon[32835]: pgmap v1704: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:24.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:24.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:24.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:24.350 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:24.454 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:24.454 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:24.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:24.462 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:24.463 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:24.463 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:24.463 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:24.463 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:24.480 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:24.480 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:24.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:24.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:24.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:24.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:24.481 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:24.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:24.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:24.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:24.773 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:21:24.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:24.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4125005869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:24.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:24.907 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4125005869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:24.907 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:24.924 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:24.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:21:25.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:25.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:25.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4125005869' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:25.101 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:25.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:25.177 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:25.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:25.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:25.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:25.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:25.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:25.570 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:25.570 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:25.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:25.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:25.940 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:25.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:25.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: pgmap v1705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:25 smithi107 ceph-mon[32381]: osdmap e568: 12 total, 3 up, 11 in 2024-08-06T18:21:26.000 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:26.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: pgmap v1705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:26.098 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:26.099 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:25 smithi117 ceph-mon[32835]: osdmap e568: 12 total, 3 up, 11 in 2024-08-06T18:21:26.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:26.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: pgmap v1705: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1333543328' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:25 smithi086 ceph-mon[29229]: osdmap e568: 12 total, 3 up, 11 in 2024-08-06T18:21:26.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:26.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:26.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:26.497 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:26.497 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:26.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:26.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:26.756 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:26.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:27.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:27.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:27.269 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:27.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:27.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:27.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:27.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:21:27.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:27.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:21:27.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:27.550 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:27.550 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:27.609 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:27.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:27.697 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:27 smithi117 ceph-mon[32835]: pgmap v1707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:27.698 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:27 smithi117 ceph-mon[32835]: osdmap e569: 12 total, 3 up, 11 in 2024-08-06T18:21:27.754 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:27.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:27.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:27 smithi086 ceph-mon[29229]: pgmap v1707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:27.828 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:27 smithi086 ceph-mon[29229]: osdmap e569: 12 total, 3 up, 11 in 2024-08-06T18:21:27.843 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:27.843 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":569,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:26.658527+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:27.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:27.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:21:27.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:27 smithi107 ceph-mon[32381]: pgmap v1707: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:27.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:27 smithi107 ceph-mon[32381]: osdmap e569: 12 total, 3 up, 11 in 2024-08-06T18:21:28.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:28.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:28.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:28.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:28.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:28.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:28.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:28.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:28.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/726793219' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:28.868 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/726793219' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:28.868 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:28.869 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:28.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:29.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:29.026 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/726793219' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:29.082 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:29.082 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:29.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:29.202 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:29.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:29.316 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:29.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:29.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:29.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:29.752 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:29 smithi086 ceph-mon[29229]: pgmap v1709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:29.756 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:29.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:21:29.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:29 smithi117 ceph-mon[32835]: pgmap v1709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:29.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:29 smithi107 ceph-mon[32381]: pgmap v1709: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:30.002 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:30.004 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:30.005 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:30.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:30.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:30.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:30.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:30.341 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:30.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:30.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:30.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:30.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:30.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:30.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:31.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:31.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:31.157 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:31.157 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:31.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:31.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:31.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:31.542 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:31.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:31.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:31.688 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:21:31.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:31 smithi107 ceph-mon[32381]: pgmap v1710: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:31.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:31.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:31.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:31.824 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:31 smithi117 ceph-mon[32835]: pgmap v1710: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:31.824 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:31.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:31 smithi086 ceph-mon[29229]: pgmap v1710: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:31.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:31.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:32.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:32.117 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:32.118 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:32.119 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:32.154 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:32.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:21:32.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:32.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:32.403 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:32.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:32.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:32.713 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:32.714 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:32.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:32.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:32.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: osdmap e570: 12 total, 3 up, 11 in 2024-08-06T18:21:32.714 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3359118788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:32.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:32.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: osdmap e570: 12 total, 3 up, 11 in 2024-08-06T18:21:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3359118788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:33.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:33.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1808385405' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: osdmap e570: 12 total, 3 up, 11 in 2024-08-06T18:21:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3359118788' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:33.228 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:33.228 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:33.237 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:33.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:33.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:33.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:33.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:33.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:33.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.810 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:33 smithi107 ceph-mon[32381]: pgmap v1712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:33.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:33.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:33.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:33 smithi086 ceph-mon[29229]: pgmap v1712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:34.021 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:21:34.021 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:21:34.021 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:21:34.021 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:21:34.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:33 smithi117 ceph-mon[32835]: pgmap v1712: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:34.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:34.208 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:34.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:34.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:34.209 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:34.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:34.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:34.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:34.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:34.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:21:34.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:21:34.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:34.864 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:34.864 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:34.965 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:34.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:35.046 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:35.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:35.179 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:35.179 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":570,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:31.669229+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:35.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:21:35.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:35.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:35.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:35.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:35.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:35.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:35 smithi086 ceph-mon[29229]: pgmap v1713: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:35.808 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3185938928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:35.889 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:35.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:35 smithi107 ceph-mon[32381]: pgmap v1713: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:35.911 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3185938928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:36.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:36.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:35 smithi117 ceph-mon[32835]: pgmap v1713: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:36.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3185938928' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:36.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:36.287 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:36.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:36.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:36.574 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:21:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:21:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:21:36.574 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:21:36.574 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:21:36.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:36.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:36.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:36.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:36.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:37.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:37.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:37.220 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:37.337 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:37.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:37.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:37.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:37.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:21:37.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:37.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:37.672 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:37.721 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:21:37.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:37 smithi086 ceph-mon[29229]: pgmap v1714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:37.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:37 smithi086 ceph-mon[29229]: osdmap e571: 12 total, 3 up, 11 in 2024-08-06T18:21:37.722 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:37.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:37 smithi117 ceph-mon[32835]: pgmap v1714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:37 smithi117 ceph-mon[32835]: osdmap e571: 12 total, 3 up, 11 in 2024-08-06T18:21:37.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:37.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:37.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:37.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:37.942 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:37 smithi107 ceph-mon[32381]: pgmap v1714: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:37.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:37 smithi107 ceph-mon[32381]: osdmap e571: 12 total, 3 up, 11 in 2024-08-06T18:21:37.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:21:38.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:38.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:38.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:38.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:38.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:38.395 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:38.395 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:38.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:38.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:38.663 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:38.663 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:38.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:38.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:38.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:38.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:38 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:38.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:38.854 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:38 smithi107 ceph-mon[32381]: osdmap e572: 12 total, 3 up, 11 in 2024-08-06T18:21:38.961 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:38.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:38 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:38.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:38.977 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:38 smithi086 ceph-mon[29229]: osdmap e572: 12 total, 3 up, 11 in 2024-08-06T18:21:38.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:38 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:38.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/734015410' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:21:38.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:38 smithi117 ceph-mon[32835]: osdmap e572: 12 total, 3 up, 11 in 2024-08-06T18:21:39.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:39.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:39.298 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:21:39.298 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:21:39.298 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:21:39.298 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:21:39.298 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.894965157000001, 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000266113, 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:21:39.299 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404599+0000", 2024-08-06T18:21:39.300 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.023e-06 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404602+0000", 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.534e-06 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404616+0000", 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3881e-05 2024-08-06T18:21:39.301 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404626+0000", 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0402e-05 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404697+0000", 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0785999999999996e-05 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.302 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404716+0000", 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9258000000000001e-05 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404864+0000", 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147229 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:21:39.303 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 73.894190577000003, 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014078100000000001, 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:21:39.304 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.305 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405373+0000", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4000000000000001e-07 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405377+0000", 2024-08-06T18:21:39.306 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.391e-06 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405387+0000", 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0040999999999999e-05 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.307 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405393+0000", 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.976e-06 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405481+0000", 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7995e-05 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:21:39.308 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405490+0000", 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1649999999999995e-06 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405513+0000", 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2572999999999999e-05 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:21:39.309 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:21:39.310 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:21:39.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:39.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:39.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:39.740 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:39.741 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:39.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:39.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:39.759 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:39.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:39.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:39.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:39.773 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:39.778 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:39.778 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:39 smithi086 ceph-mon[29229]: pgmap v1717: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:39.778 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3357559963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:39.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:39.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:39.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:39.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:39.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:39.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:39.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:39.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:39 smithi107 ceph-mon[32381]: pgmap v1717: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:39.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3357559963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:39.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:21:40.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:40.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:39 smithi117 ceph-mon[32835]: pgmap v1717: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:40.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3357559963' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:40.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:40.237 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:40.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:40.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:40.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:40.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:40.561 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:40.606 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:40.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:40.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:40.731 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:40.731 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:40.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:40.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:40.957 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:40.959 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:40.959 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:40.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:40.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:40.959 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:41.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:41.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:41.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:41.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:41.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:41.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:21:41.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:41.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:41.584 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:41.597 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:41.597 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:41.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:41.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:41.774 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:41 smithi086 ceph-mon[29229]: pgmap v1718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:41.775 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:41 smithi086 ceph-mon[29229]: osdmap e573: 12 total, 3 up, 11 in 2024-08-06T18:21:41.828 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:41.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:41 smithi107 ceph-mon[32381]: pgmap v1718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:41.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:41 smithi107 ceph-mon[32381]: osdmap e573: 12 total, 3 up, 11 in 2024-08-06T18:21:42.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:42.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:42.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:42.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:42.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:41 smithi117 ceph-mon[32835]: pgmap v1718: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:41 smithi117 ceph-mon[32835]: osdmap e573: 12 total, 3 up, 11 in 2024-08-06T18:21:42.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:42.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:42.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:42.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:42.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:42.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:42.563 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:42.564 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:42.568 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:42.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:21:42.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:21:42.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:42.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:42.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:42.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:42.818 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:42.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:42 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:21:42.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:42 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:21:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:42.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:42.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:43.052 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:43.052 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:43.053 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:42 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:21:43.053 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:42 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:21:43.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:42 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:21:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:42 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:21:43.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:43.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:43.361 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:43.361 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:43.423 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:43.423 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":573,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:41.662671+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:43.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:43.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:43.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:43.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:43.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:43.869 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:43 smithi086 ceph-mon[29229]: pgmap v1720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:43.870 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/363783452' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:43.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:43.989 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:43.989 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:43.990 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:43 smithi107 ceph-mon[32381]: pgmap v1720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:43.990 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/363783452' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:44.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:44.033 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:43 smithi117 ceph-mon[32835]: pgmap v1720: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:44.033 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/363783452' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:44.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:44.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:44.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:44.482 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:44.509 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:44.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:44.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:44.744 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:21:44.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:44.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:44.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:45.046 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:45.072 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:45.073 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:45.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:21:45.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:45.361 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:21:45.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:45.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:45.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:45.696 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:45.697 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:45.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:45.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:45.950 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:45 smithi086 ceph-mon[29229]: pgmap v1721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:45.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:45 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:45.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:45.951 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:45 smithi086 ceph-mon[29229]: osdmap e574: 12 total, 3 up, 11 in 2024-08-06T18:21:46.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:45 smithi107 ceph-mon[32381]: pgmap v1721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:45 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:45 smithi107 ceph-mon[32381]: osdmap e574: 12 total, 3 up, 11 in 2024-08-06T18:21:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:45 smithi117 ceph-mon[32835]: pgmap v1721: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:45 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:21:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/521297250' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:21:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:45 smithi117 ceph-mon[32835]: osdmap e574: 12 total, 3 up, 11 in 2024-08-06T18:21:46.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:46.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:46.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:46.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:46.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:46.752 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:46.753 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:46.852 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:46.852 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:46.869 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:46.870 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:46.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:46.886 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:46.887 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:46.888 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:46.922 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:47.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:47.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:47.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:47.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:47.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:47.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:47.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:47.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:47.671 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:47.704 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:47.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:21:47.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:47.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:47.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:47 smithi086 ceph-mon[29229]: pgmap v1723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:47 smithi086 ceph-mon[29229]: osdmap e575: 12 total, 3 up, 11 in 2024-08-06T18:21:47.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/286119890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:47.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:47.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:47 smithi117 ceph-mon[32835]: pgmap v1723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:47.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:47 smithi117 ceph-mon[32835]: osdmap e575: 12 total, 3 up, 11 in 2024-08-06T18:21:47.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/286119890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:47.957 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.957 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:47 smithi107 ceph-mon[32381]: pgmap v1723: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:47 smithi107 ceph-mon[32381]: osdmap e575: 12 total, 3 up, 11 in 2024-08-06T18:21:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/286119890' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:47.971 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:48.067 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:48.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:48.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:48.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:48.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:48.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:21:48.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:48.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:48.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:48 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:21:48.810 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:48 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:21:48.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:48 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:21:48.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:48 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:21:48.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:48.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:48 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:21:48.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:48 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:21:48.963 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:48.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:49.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:49.247 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:49.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:49.498 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:49.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:49.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:21:49.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:49.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:49.881 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:49.881 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":575,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:46.664145+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:49.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:49.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:49 smithi086 ceph-mon[29229]: pgmap v1725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:49.912 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:49.912 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:49 smithi107 ceph-mon[32381]: pgmap v1725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:50.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:50.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:49 smithi117 ceph-mon[32835]: pgmap v1725: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:50.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:50.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:21:50.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:50.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:50.334 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:50.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:50.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:50.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1439946549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:21:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:21:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:21:50.713 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:21:50.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1439946549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:21:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:21:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:21:50.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:21:50.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1439946549' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:50.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:21:50.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:21:50.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:21:50.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:21:50.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:51.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:51.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:51.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:51.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:51.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:51.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:51.807 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:51 smithi086 ceph-mon[29229]: pgmap v1726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:51.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:51.871 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:51 smithi107 ceph-mon[32381]: pgmap v1726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:51.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:52.021 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:51 smithi117 ceph-mon[32835]: pgmap v1726: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:52.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:52.153 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:52.153 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:52.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:52.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:52.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:21:52.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:52.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:52.681 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:21:52.681 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:52.681 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:52.681 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:52.681 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:52.699 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:52.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:52.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:52.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:53.007 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:53.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:53.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:53.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:21:53.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:53.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:53.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:53.317 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:53.317 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:53.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:53.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:53.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:53.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:53.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:53.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:53.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:53.731 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:21:53.731 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:21:53.731 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:21:53.748 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:21:53.748 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:21:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:21:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:21:53.749 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:21:53.766 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:21:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:21:53.767 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:21:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:21:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:21:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:21:53.768 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:21:53.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:53 smithi117 ceph-mon[32835]: pgmap v1727: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:53 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:53 smithi117 ceph-mon[32835]: osdmap e576: 12 total, 3 up, 11 in 2024-08-06T18:21:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:53 smithi107 ceph-mon[32381]: pgmap v1727: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:53 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:53 smithi107 ceph-mon[32381]: osdmap e576: 12 total, 3 up, 11 in 2024-08-06T18:21:53.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:54.098 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:54.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:53 smithi086 ceph-mon[29229]: pgmap v1727: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:53 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:21:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3831963163' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:21:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:53 smithi086 ceph-mon[29229]: osdmap e576: 12 total, 3 up, 11 in 2024-08-06T18:21:54.244 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:21:54.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:21:54.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:54.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:54.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:54.531 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:54.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:54.648 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:54.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:21:54.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:54.792 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/547549829' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:54.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/547549829' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.896 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:54.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:21:54.932 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:21:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/547549829' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:21:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:54.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:21:55.000 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:55.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:55.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:55.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:55.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:55.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:55.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:55.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:55.375 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:55.375 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:55.435 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:55.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:55.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:55.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:55 smithi107 ceph-mon[32381]: pgmap v1729: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:55.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:55.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:55.969 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:55.991 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:55 smithi117 ceph-mon[32835]: pgmap v1729: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:56.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:55 smithi086 ceph-mon[29229]: pgmap v1729: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:56.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:56.100 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:56.100 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:56.101 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:56.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:56.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:56.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:21:56.287 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:56.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:56.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:56.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:56.645 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:56.645 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:56.706 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:56.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:56.846 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:21:56.846 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":577,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:21:56.666816+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:21:56.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:21:56.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:21:57.033 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:57.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:21:57.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:57.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:57.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:21:57.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:21:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:21:57.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:57.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:57.277 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:57.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:57.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:57.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:57.600 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:21:57.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:57.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:21:57.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:57.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:57 smithi086 ceph-mon[29229]: pgmap v1730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:57 smithi086 ceph-mon[29229]: osdmap e577: 12 total, 3 up, 11 in 2024-08-06T18:21:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/510237771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:57.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:57 smithi107 ceph-mon[32381]: pgmap v1730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:57 smithi107 ceph-mon[32381]: osdmap e577: 12 total, 3 up, 11 in 2024-08-06T18:21:57.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/510237771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:57.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:58.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:58.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:57 smithi117 ceph-mon[32835]: pgmap v1730: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:58.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:57 smithi117 ceph-mon[32835]: osdmap e577: 12 total, 3 up, 11 in 2024-08-06T18:21:58.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/510237771' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:21:58.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:21:58.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:58.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:58.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:58.675 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:58.676 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:58.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:58.814 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:21:58.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:21:59.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:21:59.053 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:59.053 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:59.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:59.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:21:59.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:21:59.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:21:59.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:21:59.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:21:59.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:21:59.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:21:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:21:59.583 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:21:59.583 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:21:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:21:59.671 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:21:59.688 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:21:59.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:21:59.714 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:59 smithi117 ceph-mon[32835]: pgmap v1732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:59.715 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:21:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:59.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:21:59.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:59 smithi086 ceph-mon[29229]: pgmap v1732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:59.845 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:21:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:21:59.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:21:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:59 smithi107 ceph-mon[32381]: pgmap v1732: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:21:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:21:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:00.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:00.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:00.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:00.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:00.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:00.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:00.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:00.747 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:00.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:00.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:00 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:00.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:00.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:00 smithi107 ceph-mon[32381]: osdmap e578: 12 total, 3 up, 11 in 2024-08-06T18:22:01.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:01.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:00 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:01.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:01.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:00 smithi086 ceph-mon[29229]: osdmap e578: 12 total, 3 up, 11 in 2024-08-06T18:22:01.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:00 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:01.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2417942379' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:01.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:00 smithi117 ceph-mon[32835]: osdmap e578: 12 total, 3 up, 11 in 2024-08-06T18:22:01.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:22:01.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:01.176 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:01.176 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:01.210 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:01.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:01.380 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:01.381 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:01.400 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:01.401 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:01.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:01.422 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:01.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:01.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:01.635 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:01.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:01.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:01 smithi107 ceph-mon[32381]: pgmap v1734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:01.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3804032462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:01.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:01 smithi107 ceph-mon[32381]: osdmap e579: 12 total, 3 up, 11 in 2024-08-06T18:22:01.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:01 smithi117 ceph-mon[32835]: pgmap v1734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3804032462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:01.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:01 smithi117 ceph-mon[32835]: osdmap e579: 12 total, 3 up, 11 in 2024-08-06T18:22:01.934 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:01.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:22:01.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:02.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:01 smithi086 ceph-mon[29229]: pgmap v1734: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 327 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:02.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3804032462' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:02.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:01 smithi086 ceph-mon[29229]: osdmap e579: 12 total, 3 up, 11 in 2024-08-06T18:22:02.088 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:02.108 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:02.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:02.179 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:02.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:02.242 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:02.242 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:02.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:02.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:02.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:02.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:02.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:02.749 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:02.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:02.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:22:02.902 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.902 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.902 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:02.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:03.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:03.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:03.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:03.389 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:03.410 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:03.478 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:03.478 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:03.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:03.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:03.675 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:03.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:03.826 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:03 smithi086 ceph-mon[29229]: pgmap v1736: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:03.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:03 smithi107 ceph-mon[32381]: pgmap v1736: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:04.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:04.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:04.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:04.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:22:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:03 smithi117 ceph-mon[32835]: pgmap v1736: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:04.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:04.332 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:04.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:22:04.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:04.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:04.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:04.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:04.587 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:04.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:04.652 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:22:04.652 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:04.652 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:04.653 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":579,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:22:01.668133+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:58.279909+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:22:04.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:04.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:04.915 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2830809743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:04.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2830809743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:04.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:04.988 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2830809743' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:05.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:05.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:05.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:05.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:05.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:05.700 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:05.700 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:05.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:05.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:05.855 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:05 smithi086 ceph-mon[29229]: pgmap v1737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:05 smithi107 ceph-mon[32381]: pgmap v1737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:05.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:05.987 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:05 smithi117 ceph-mon[32835]: pgmap v1737: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:06.024 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:06.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:06.112 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:06.113 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:06.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:06.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:06.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:06.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:06.618 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:06.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:22:06.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:06.669 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:22:06.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:06.868 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:06.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:06.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:22:06.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:22:06.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: osdmap e580: 12 total, 3 up, 11 in 2024-08-06T18:22:06.921 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:06.922 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:06.922 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:06.922 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:06 smithi086 ceph-mon[29229]: osdmap e581: 12 total, 3 up, 11 in 2024-08-06T18:22:06.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:06.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: osdmap e580: 12 total, 3 up, 11 in 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:06.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:06 smithi107 ceph-mon[32381]: osdmap e581: 12 total, 3 up, 11 in 2024-08-06T18:22:06.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:22:06.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:22:06.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: osdmap e580: 12 total, 3 up, 11 in 2024-08-06T18:22:06.984 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:06.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:06.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1081914831' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:06.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:06 smithi117 ceph-mon[32835]: osdmap e581: 12 total, 3 up, 11 in 2024-08-06T18:22:07.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:07.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:07.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:07.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:07.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:07.488 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:07.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:07.677 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:07.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:07.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:07 smithi117 ceph-mon[32835]: pgmap v1739: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:07.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:07 smithi086 ceph-mon[29229]: pgmap v1739: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:07.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:07 smithi107 ceph-mon[32381]: pgmap v1739: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:08.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:08.198 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:08.233 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:08.233 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:08.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:08.459 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:08.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:08.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:08.612 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:08.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:08.613 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:08.630 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:08.630 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:08.630 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:08.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:08.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:08.648 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:08.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:08.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:08.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:08.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:08.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:08.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:08.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:22:08.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:08.763 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/409265711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:08.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:09.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:09.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/409265711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:09.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/409265711' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:09.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:09.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:09.123 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:09.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:22:09.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:09.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:09.371 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:09.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:09.480 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:09.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:09.485 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:09.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:22:09.536 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:09.537 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:09.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:09.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:09.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:09 smithi117 ceph-mon[32835]: pgmap v1741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:09.766 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.767 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.767 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.793 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:09 smithi086 ceph-mon[29229]: pgmap v1741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:09.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.794 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:09.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:09.997 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:09 smithi107 ceph-mon[32381]: pgmap v1741: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:09.997 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.997 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:09.997 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:10.099 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:10.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:10.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:10.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:10.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:10.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:10.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:10.632 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:10.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:10.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:10.981 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:11.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:11.055 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:11.055 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:11.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:11.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:11.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:11.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:22:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:11.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:22:11.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:11.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:11.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:11.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:11.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:11.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:11.791 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:11.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:11.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:11 smithi117 ceph-mon[32835]: pgmap v1742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:11.864 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:22:11.865 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":581,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:22:06.670074+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5804032,"flags_set":["nodeep-scrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:20:53.803675+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:05.534300+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:22:11.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:11.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:11 smithi086 ceph-mon[29229]: pgmap v1742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:12.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:12.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:11 smithi107 ceph-mon[32381]: pgmap v1742: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:12.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:12.236 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:12.236 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:12.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:12.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:12.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:12.765 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:12.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:12.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1068880630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:12.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:13.026 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:13.052 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1068880630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:13.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1068880630' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:13.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:13.200 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:13.200 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:13.297 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:13.318 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:13.378 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:13.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:13.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:13.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:13.729 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:13 smithi086 ceph-mon[29229]: pgmap v1743: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:13.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:13.731 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:22:13.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:13 smithi117 ceph-mon[32835]: pgmap v1743: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:13.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:14.026 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:14.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:22:14.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:13 smithi107 ceph-mon[32381]: pgmap v1743: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:14.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:14.275 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:22:14.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:14.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:14.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:14.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:14.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:14.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:14.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: osdmap e582: 12 total, 3 up, 11 in 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:22:14.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:22:14.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:15.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: osdmap e582: 12 total, 3 up, 11 in 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:22:15.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:22:15.060 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:22:15.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:15.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1181663579' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: osdmap e582: 12 total, 3 up, 11 in 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:22:15.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:22:15.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:15.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:15.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:15.498 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:15.498 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:15.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:15.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:15.740 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:15.741 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:15.741 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:15.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:15.758 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:15.759 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1430418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:15.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:15.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:15.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:15.776 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:15.777 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:15.827 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:15.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:15.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:16.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:22:16.082 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:16.083 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:16.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:16.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1430418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1430418016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:16.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:16.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:16.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:16.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:16.385 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:16.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:22:16.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:16.501 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:16.501 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:16.543 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:16.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:16.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:16.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:16.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.931 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:16 smithi086 ceph-mon[29229]: osdmap e583: 12 total, 3 up, 11 in 2024-08-06T18:22:16.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:16.932 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:16 smithi107 ceph-mon[32381]: osdmap e583: 12 total, 3 up, 11 in 2024-08-06T18:22:17.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:17.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:17.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:17.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:17.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:16 smithi117 ceph-mon[32835]: osdmap e583: 12 total, 3 up, 11 in 2024-08-06T18:22:17.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:17.274 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:17.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:17.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:17.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:22:17.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:17.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:17.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:17.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:17.740 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:17.740 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:17.827 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:17 smithi086 ceph-mon[29229]: pgmap v1746: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:17.827 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:17.827 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:17.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:17.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:18.004 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:17 smithi107 ceph-mon[32381]: pgmap v1746: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:17 smithi117 ceph-mon[32835]: pgmap v1746: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:18.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:18.262 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:18.262 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:18.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:18.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:18.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:22:18.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:18.548 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:18.548 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:18.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:18.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:18.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:18.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:18.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:19.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:19.124 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:19.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:19.193 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:19.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:19.311 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:22:19.311 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":584,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:22:18.742668+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:18.579693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:05.534300+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:22:19.313 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:19.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:19.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:19.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:19.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:19.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:19 smithi117 ceph-mon[32835]: pgmap v1748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:19.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:19 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:22:19.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:19 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:22:19.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:19 smithi117 ceph-mon[32835]: osdmap e584: 12 total, 3 up, 11 in 2024-08-06T18:22:19.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3741538561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:19.932 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:19.932 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:19.932 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:19 smithi086 ceph-mon[29229]: pgmap v1748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:19.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:19 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:22:19.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:19 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:22:19.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:19 smithi086 ceph-mon[29229]: osdmap e584: 12 total, 3 up, 11 in 2024-08-06T18:22:19.933 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3741538561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:19.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:20.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:19 smithi107 ceph-mon[32381]: pgmap v1748: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:20.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:19 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:22:20.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:19 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:22:20.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:19 smithi107 ceph-mon[32381]: osdmap e584: 12 total, 3 up, 11 in 2024-08-06T18:22:20.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3741538561' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:20.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:20.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:20.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:20.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:20.568 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:20.569 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:20.667 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:20.795 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:20.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:20.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:20.992 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:21.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:21.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:21.217 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:21.217 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:21.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:21.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:21.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:21.682 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:21.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:22:21.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:21.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:21.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:21 smithi086 ceph-mon[29229]: pgmap v1750: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:21.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:21 smithi086 ceph-mon[29229]: osdmap e585: 12 total, 3 up, 11 in 2024-08-06T18:22:21.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:22.021 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:21 smithi117 ceph-mon[32835]: pgmap v1750: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:22.021 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:21 smithi117 ceph-mon[32835]: osdmap e585: 12 total, 3 up, 11 in 2024-08-06T18:22:22.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:21 smithi107 ceph-mon[32381]: pgmap v1750: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:21 smithi107 ceph-mon[32381]: osdmap e585: 12 total, 3 up, 11 in 2024-08-06T18:22:22.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:22.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:22.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:22.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:22.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:22.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:22.653 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:22.789 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:22.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:23.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:23.185 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:23.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:23.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:23.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:22:23.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:23.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:23.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:23.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:23.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:23.863 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:23.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:23 smithi086 ceph-mon[29229]: pgmap v1752: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:23.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:23.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:23.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:23 smithi117 ceph-mon[32835]: pgmap v1752: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:24.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:24.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:23 smithi107 ceph-mon[32381]: pgmap v1752: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:24.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:24.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:24.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:24.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:24.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:24.682 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:24.683 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:24.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:24.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:24.971 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:24.971 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:25.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:25.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:25.273 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:25.273 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:25.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:25.357 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:25.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:25.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:25.730 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:25.731 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:25.731 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:25.747 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:25.748 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:25.748 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:25.748 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:25.748 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:25.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:25.766 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:25.766 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:25.766 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:25.767 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:25.779 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:25 smithi107 ceph-mon[32381]: pgmap v1753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:25.779 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2631786305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:25.780 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:25.780 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:25.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:25.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:25 smithi086 ceph-mon[29229]: pgmap v1753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:25.911 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2631786305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:26.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:25 smithi117 ceph-mon[32835]: pgmap v1753: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2631786305' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:26.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:26.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:26.421 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:26.460 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:26.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:26.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:26.759 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:26.760 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:26.760 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:26.760 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:26.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:26.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.015 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:27.015 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:27.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:27.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:27.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:27.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:27.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:27.489 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:27.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:27.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:27.782 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:27 smithi117 ceph-mon[32835]: pgmap v1754: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:27.812 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:27 smithi086 ceph-mon[29229]: pgmap v1754: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:27.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:27.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:27.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:28.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:28.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:27 smithi107 ceph-mon[32381]: pgmap v1754: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:28.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:28.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:28.434 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:28.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:22:28.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:22:28.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:28.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:28.763 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:28.763 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:28.769 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:22:28.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:22:28.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:29.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:29.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:22:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:22:29.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:29.258 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:29.258 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:29.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:29.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:29.556 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:29.595 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:29.634 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:29.770 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:29.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:29.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:29 smithi117 ceph-mon[32835]: pgmap v1755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:29.932 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:29 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:29.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:22:29.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:29 smithi117 ceph-mon[32835]: osdmap e586: 12 total, 3 up, 11 in 2024-08-06T18:22:30.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:29 smithi086 ceph-mon[29229]: pgmap v1755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:30.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:29 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:30.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:22:30.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:29 smithi086 ceph-mon[29229]: osdmap e586: 12 total, 3 up, 11 in 2024-08-06T18:22:30.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:30.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:29 smithi107 ceph-mon[32381]: pgmap v1755: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:29 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2313024249' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:22:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:29 smithi107 ceph-mon[32381]: osdmap e586: 12 total, 3 up, 11 in 2024-08-06T18:22:30.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:30.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:30.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:30.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:30.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:30.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:31.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:31.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:31.288 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:22:31.288 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":586,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:22:28.764902+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5801984,"flags_set":["noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:18.579693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:05.534300+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:21:20.511907+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:22:31.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:31.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:31.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:22:31.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:31.782 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:31.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:31 smithi086 ceph-mon[29229]: pgmap v1757: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4198625859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:31.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:31 smithi086 ceph-mon[29229]: osdmap e587: 12 total, 3 up, 11 in 2024-08-06T18:22:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:31 smithi117 ceph-mon[32835]: pgmap v1757: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4198625859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:31.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:31 smithi117 ceph-mon[32835]: osdmap e587: 12 total, 3 up, 11 in 2024-08-06T18:22:31.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:31 smithi107 ceph-mon[32381]: pgmap v1757: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4198625859' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:31 smithi107 ceph-mon[32381]: osdmap e587: 12 total, 3 up, 11 in 2024-08-06T18:22:32.047 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:32.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:32.264 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:32.264 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:32.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:32.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:32.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:32.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:32.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:32.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:32.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:32.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:33.112 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:33.112 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:33.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:33.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:33.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:33.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:33.547 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:33.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:33.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:33.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:33.953 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:33 smithi107 ceph-mon[32381]: pgmap v1759: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:33.953 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:33.954 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:34.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:34.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:33 smithi086 ceph-mon[29229]: pgmap v1759: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:33 smithi117 ceph-mon[32835]: pgmap v1759: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:34.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:34.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:34.255 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:34.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:22:34.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:34.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:34.528 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:34.529 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:34.529 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:34.533 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:34.542 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:34.542 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:34.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:34.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:34.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:34.561 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:34.562 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:34.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:34.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:34.562 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:34.563 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:34.563 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:34.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:34.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:34.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:34.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3624447761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:34.999 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:35.005 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3624447761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:35.035 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:35.035 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:35.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3624447761' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:35.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:35.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:35.292 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:35.292 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:35.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:35.412 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:35.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:35.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:35.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:35.672 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:35.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:35.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd dump --format=json 2024-08-06T18:22:35.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:35 smithi086 ceph-mon[29229]: pgmap v1760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:35.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:35.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:35.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:35.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:36.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:36.098 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:35 smithi107 ceph-mon[32381]: pgmap v1760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:36.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:35 smithi117 ceph-mon[32835]: pgmap v1760: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:36.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:36.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:36.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:36.549 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:36.549 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:36.575 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:36.576 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:36.610 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:36.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:36.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:37.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:37.051 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:37.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:37.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:37.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:37.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:37.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:37.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:37.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:37.845 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:37 smithi117 ceph-mon[32835]: pgmap v1761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:37.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:37 smithi107 ceph-mon[32381]: pgmap v1761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:38.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:37 smithi086 ceph-mon[29229]: pgmap v1761: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:38.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:38.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:38.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:38.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:38.577 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:22:38.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:38.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:38.793 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:22:38.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:39.036 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:39.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:38 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:22:39.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:38 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:22:39.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:38 smithi107 ceph-mon[32381]: osdmap e588: 12 total, 3 up, 11 in 2024-08-06T18:22:39.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:39.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:38 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:22:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:38 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:22:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:38 smithi086 ceph-mon[29229]: osdmap e588: 12 total, 3 up, 11 in 2024-08-06T18:22:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:38 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:22:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:38 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:22:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:38 smithi117 ceph-mon[32835]: osdmap e588: 12 total, 3 up, 11 in 2024-08-06T18:22:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:39.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:39.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:39.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:39.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:39.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:39.763 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:22:39.763 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:22:39.764 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:22:39.764 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:22:39.793 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:39.794 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:39.794 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:39 smithi107 ceph-mon[32381]: pgmap v1763: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:39.794 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:39 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:39.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:39.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:40.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:40.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:40.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:39 smithi086 ceph-mon[29229]: pgmap v1763: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:40.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:39 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:40.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:40.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:39 smithi086 ceph-mon[29229]: osdmap e589: 12 total, 3 up, 11 in 2024-08-06T18:22:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:39 smithi117 ceph-mon[32835]: pgmap v1763: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:39 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:39 smithi117 ceph-mon[32835]: osdmap e589: 12 total, 3 up, 11 in 2024-08-06T18:22:40.133 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3526066229' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:22:40.133 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:39 smithi107 ceph-mon[32381]: osdmap e589: 12 total, 3 up, 11 in 2024-08-06T18:22:40.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:40.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:40.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:22:40.436 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:40.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:40.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:40.700 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:40.701 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:40.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:40.919 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:41.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:41.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:41.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:41.351 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:41.351 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:41.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:41.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:22:41.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:41.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:41.753 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:41.753 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:41.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:41.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:42.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:41 smithi086 ceph-mon[29229]: pgmap v1765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:42.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:41 smithi117 ceph-mon[32835]: pgmap v1765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:41 smithi107 ceph-mon[32381]: pgmap v1765: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:42.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:42.280 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:42.280 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:42.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:42.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:42.668 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:42.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:42.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:42.839 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-08-06T18:22:42.840 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":589,"fsid":"fd7e697e-541a-11ef-bccc-c7b262605968","created":"2024-08-06T17:42:16.357349+0000","modified":"2024-08-06T18:22:38.789873+0000","last_up_change":"2024-08-06T17:51:23.172779+0000","last_in_change":"2024-08-06T17:51:29.812887+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":28,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-08-06T17:45:05.808831+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"test-rados-api-smithi117-58763-1","create_time":"2024-08-06T17:50:42.731444+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}},"read_balance":{"error_message":"pool 2 has some PGs where all OSDs are with primary_affinity 0 (at least pg 2.0), read balance score may not be reliable","score_acting":0,"score_stable":0}}],"osds":[{"osd":0,"uuid":"2ab66bcc-745f-4928-8cff-7095957437c5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":116,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6802","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6803","nonce":1557268730}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6804","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6805","nonce":1557268730}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6808","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6809","nonce":1557268730}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6806","nonce":1557268730},{"type":"v1","addr":"172.21.15.86:6807","nonce":1557268730}]},"public_addr":"172.21.15.86:6803/1557268730","cluster_addr":"172.21.15.86:6805/1557268730","heartbeat_back_addr":"172.21.15.86:6809/1557268730","heartbeat_front_addr":"172.21.15.86:6807/1557268730","state":["exists","up"]},{"osd":1,"uuid":"331fb57c-4513-4398-afda-0ba9442c0a06","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":15,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6811","nonce":1859701100}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6813","nonce":1859701100}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6817","nonce":1859701100}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1859701100},{"type":"v1","addr":"172.21.15.86:6815","nonce":1859701100}]},"public_addr":"172.21.15.86:6811/1859701100","cluster_addr":"172.21.15.86:6813/1859701100","heartbeat_back_addr":"172.21.15.86:6817/1859701100","heartbeat_front_addr":"172.21.15.86:6815/1859701100","state":["exists"]},{"osd":2,"uuid":"51280800-811c-42ce-97c0-aa3787f5a681","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":21,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6818","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6819","nonce":1080990493}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6820","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6821","nonce":1080990493}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6824","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6825","nonce":1080990493}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6822","nonce":1080990493},{"type":"v1","addr":"172.21.15.86:6823","nonce":1080990493}]},"public_addr":"172.21.15.86:6819/1080990493","cluster_addr":"172.21.15.86:6821/1080990493","heartbeat_back_addr":"172.21.15.86:6825/1080990493","heartbeat_front_addr":"172.21.15.86:6823/1080990493","state":["exists","up"]},{"osd":3,"uuid":"121c9b13-fa41-4680-8424-c96f1ddcec09","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":26,"last_clean_end":98,"up_from":112,"up_thru":112,"down_at":116,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6810","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6811","nonce":1831259870}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6812","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6813","nonce":1831259870}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6816","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6817","nonce":1831259870}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.86:6814","nonce":1831259870},{"type":"v1","addr":"172.21.15.86:6815","nonce":1831259870}]},"public_addr":"172.21.15.86:6811/1831259870","cluster_addr":"172.21.15.86:6813/1831259870","heartbeat_back_addr":"172.21.15.86:6817/1831259870","heartbeat_front_addr":"172.21.15.86:6815/1831259870","state":["exists"]},{"osd":4,"uuid":"a1dd2693-1793-4b19-84da-e020a8252b4d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":32,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6800","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6801","nonce":1874359981}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6802","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6803","nonce":1874359981}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6806","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6807","nonce":1874359981}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6804","nonce":1874359981},{"type":"v1","addr":"172.21.15.107:6805","nonce":1874359981}]},"public_addr":"172.21.15.107:6801/1874359981","cluster_addr":"172.21.15.107:6803/1874359981","heartbeat_back_addr":"172.21.15.107:6807/1874359981","heartbeat_front_addr":"172.21.15.107:6805/1874359981","state":["exists"]},{"osd":5,"uuid":"27dd75cf-526a-44b5-8f5e-c53573212054","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":40,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6808","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6809","nonce":1512731119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6810","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6811","nonce":1512731119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6814","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6815","nonce":1512731119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6812","nonce":1512731119},{"type":"v1","addr":"172.21.15.107:6813","nonce":1512731119}]},"public_addr":"172.21.15.107:6809/1512731119","cluster_addr":"172.21.15.107:6811/1512731119","heartbeat_back_addr":"172.21.15.107:6815/1512731119","heartbeat_front_addr":"172.21.15.107:6813/1512731119","state":["exists"]},{"osd":6,"uuid":"26ac2f5b-7633-46a8-a433-bab4bef555d7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6816","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6817","nonce":949622377}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6818","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6819","nonce":949622377}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6822","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6823","nonce":949622377}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6820","nonce":949622377},{"type":"v1","addr":"172.21.15.107:6821","nonce":949622377}]},"public_addr":"172.21.15.107:6817/949622377","cluster_addr":"172.21.15.107:6819/949622377","heartbeat_back_addr":"172.21.15.107:6823/949622377","heartbeat_front_addr":"172.21.15.107:6821/949622377","state":["exists","up"]},{"osd":7,"uuid":"1c308454-14da-4e8c-9065-60e3306e30f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":53,"up_thru":100,"down_at":101,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6824","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6825","nonce":1970096213}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6826","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6827","nonce":1970096213}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6830","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6831","nonce":1970096213}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.107:6828","nonce":1970096213},{"type":"v1","addr":"172.21.15.107:6829","nonce":1970096213}]},"public_addr":"172.21.15.107:6825/1970096213","cluster_addr":"172.21.15.107:6827/1970096213","heartbeat_back_addr":"172.21.15.107:6831/1970096213","heartbeat_front_addr":"172.21.15.107:6829/1970096213","state":["exists"]},{"osd":8,"uuid":"db0ad585-df38-4d49-a78e-797f0aed4b1d","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6801","nonce":1266762312}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6803","nonce":1266762312}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6807","nonce":1266762312}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":1266762312},{"type":"v1","addr":"172.21.15.117:6805","nonce":1266762312}]},"public_addr":"172.21.15.117:6801/1266762312","cluster_addr":"172.21.15.117:6803/1266762312","heartbeat_back_addr":"172.21.15.117:6807/1266762312","heartbeat_front_addr":"172.21.15.117:6805/1266762312","state":["exists"]},{"osd":9,"uuid":"b35e8fd4-d68d-4704-bb06-f15928193bdb","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":99,"down_at":100,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6809","nonce":170095576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6811","nonce":170095576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6815","nonce":170095576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":170095576},{"type":"v1","addr":"172.21.15.117:6813","nonce":170095576}]},"public_addr":"172.21.15.117:6809/170095576","cluster_addr":"172.21.15.117:6811/170095576","heartbeat_back_addr":"172.21.15.117:6815/170095576","heartbeat_front_addr":"172.21.15.117:6813/170095576","state":["exists"]},{"osd":10,"uuid":"c41ae499-4a09-4ead-bcfa-fdf8838f397c","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":75,"up_thru":101,"down_at":104,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6817","nonce":678280815}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6819","nonce":678280815}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6823","nonce":678280815}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":678280815},{"type":"v1","addr":"172.21.15.117:6821","nonce":678280815}]},"public_addr":"172.21.15.117:6817/678280815","cluster_addr":"172.21.15.117:6819/678280815","heartbeat_back_addr":"172.21.15.117:6823/678280815","heartbeat_front_addr":"172.21.15.117:6821/678280815","state":["exists"]},{"osd":11,"uuid":"3f543ee4-77a1-48fb-92cc-e920c871a986","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":99,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6825","nonce":3154565136}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6827","nonce":3154565136}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6831","nonce":3154565136}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":3154565136},{"type":"v1","addr":"172.21.15.117:6829","nonce":3154565136}]},"public_addr":"172.21.15.117:6825/3154565136","cluster_addr":"172.21.15.117:6827/3154565136","heartbeat_back_addr":"172.21.15.117:6831/3154565136","heartbeat_front_addr":"172.21.15.117:6829/3154565136","state":["exists"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:18.579693+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:33.745339+0000","dead_epoch":98},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:05.534300+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":65536,"last_purged_snaps_scrub":"2024-08-06T17:50:28.891965+0000","dead_epoch":115},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:44.219034+0000","dead_epoch":98},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:23.014324+0000","dead_epoch":98},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T18:22:37.277432+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:37.118328+0000","dead_epoch":100},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:50:25.779777+0000","dead_epoch":98},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:34.567419+0000","dead_epoch":100},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.248089+0000","dead_epoch":103},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-08-06T17:49:55.840955+0000","dead_epoch":98}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.86:6801/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6800/2391299811":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/1285673635":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/3531235780":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2979281859":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/1213671535":"2024-08-07T17:43:07.505980+0000","172.21.15.86:6801/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:6800/3936694336":"2024-08-07T17:42:44.380746+0000","172.21.15.86:0/2397207804":"2024-08-07T17:43:07.505980+0000","172.21.15.86:0/4243608842":"2024-08-07T17:42:44.380746+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-08-06T18:22:42.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:43.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:43.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:43.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:43.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:43.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:43.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:43.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:43.646 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1451, in _do_thrash self.test_map_discontinuity() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1213, in test_map_discontinuity self.ceph_manager.wait_for_all_osds_up( File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2863, in wait_for_all_osds_up assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_for_all_osds_up 2024-08-06T18:22:43.647 ERROR:tasks.thrashosds.thrasher:exception: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1332, in do_thrash self._do_thrash() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1451, in _do_thrash self.test_map_discontinuity() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1213, in test_map_discontinuity self.ceph_manager.wait_for_all_osds_up( File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2863, in wait_for_all_osds_up assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_for_all_osds_up 2024-08-06T18:22:43.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:43.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:44.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:44.050 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:43 smithi107 ceph-mon[32381]: pgmap v1766: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:44.050 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3981202412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:43 smithi086 ceph-mon[29229]: pgmap v1766: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3981202412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:44.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:43 smithi117 ceph-mon[32835]: pgmap v1766: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3981202412' entity='client.admin' cmd={"prefix": "osd dump", "format": "json"} : dispatch 2024-08-06T18:22:44.151 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:44.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:44.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:44.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:44.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:44.582 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:44.604 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:44.604 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:44.605 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:44.622 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:44.623 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:44.640 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:44.641 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:44.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:44.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:44.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2950792795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:44.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2950792795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:44.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:44.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2950792795' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:44.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:44.964 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:45.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:45.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:45.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:45.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:45.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:45.714 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:22:45.714 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:22:45.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:45.884 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:46.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:46.025 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:46.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:46.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:45 smithi086 ceph-mon[29229]: pgmap v1767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:46.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.056 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:45 smithi107 ceph-mon[32381]: pgmap v1767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:46.056 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.056 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.056 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:45 smithi117 ceph-mon[32835]: pgmap v1767: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:46.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:46.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:46.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:46.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:22:46.562 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:46.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:46.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:46.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:47.007 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:47.008 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:47.085 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:47.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:47.099 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:47.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:47.320 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:47.320 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:47.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:47.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:47.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:47.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:47.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:47.925 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:47 smithi086 ceph-mon[29229]: pgmap v1768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:47.925 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:47.925 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:48.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:48.059 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:47 smithi107 ceph-mon[32381]: pgmap v1768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:47 smithi117 ceph-mon[32835]: pgmap v1768: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:48.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:48.299 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:48.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:48.568 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:48.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:48.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:48.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:49.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:22:49.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:49.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:49.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:49.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:49.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:49.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:49.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:49 smithi086 ceph-mon[29229]: pgmap v1769: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:49.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:49 smithi107 ceph-mon[32381]: pgmap v1769: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:49.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:49.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:50.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:50.077 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:22:50.077 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:22:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:49 smithi117 ceph-mon[32835]: pgmap v1769: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:50.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:50.322 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:50.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:50.484 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:50.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:50.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:50.737 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:50.737 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:50.818 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:22:50.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:50.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:50.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:51.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:51.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:22:51.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:51.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:51.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:51.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:51.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:51.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:51.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:51.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:51.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:51.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:51.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:51.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:22:51.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:52.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:52.103 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:52.104 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:52.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:51 smithi086 ceph-mon[29229]: pgmap v1770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:52.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:51 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:52.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:52.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:51 smithi086 ceph-mon[29229]: osdmap e590: 12 total, 3 up, 11 in 2024-08-06T18:22:52.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:51 smithi086 ceph-mon[29229]: osdmap e591: 12 total, 3 up, 11 in 2024-08-06T18:22:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:51 smithi117 ceph-mon[32835]: pgmap v1770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:51 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:51 smithi117 ceph-mon[32835]: osdmap e590: 12 total, 3 up, 11 in 2024-08-06T18:22:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:51 smithi117 ceph-mon[32835]: osdmap e591: 12 total, 3 up, 11 in 2024-08-06T18:22:52.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:52.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:51 smithi107 ceph-mon[32381]: pgmap v1770: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:51 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:22:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1266836544' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:22:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:51 smithi107 ceph-mon[32381]: osdmap e590: 12 total, 3 up, 11 in 2024-08-06T18:22:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:51 smithi107 ceph-mon[32381]: osdmap e591: 12 total, 3 up, 11 in 2024-08-06T18:22:52.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:52.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:52.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:52.610 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:52.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:52.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:52.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:52.809 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:52.879 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:53.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:53.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:53.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:53.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:53.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:53.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:53.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:53.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:53.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:53.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:22:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:53 smithi107 ceph-mon[32381]: pgmap v1773: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/896397811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:54.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:54.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:53 smithi086 ceph-mon[29229]: pgmap v1773: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:54.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/896397811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:54.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:53 smithi117 ceph-mon[32835]: pgmap v1773: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/896397811' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:22:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:54.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:54.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:54.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:54.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:54.397 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:22:54.398 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:22:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:22:54.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:22:54.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:54.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:54.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:54.887 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:54.959 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:54.959 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:54.960 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:54.960 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:54.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:54.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:54.985 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:22:54.985 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:22:55.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:55.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:22:55.255 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:55.255 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:55.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:55.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:55.474 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:22:55.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:55.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:55.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:55.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:55 smithi107 ceph-mon[32381]: pgmap v1774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:55.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:55.993 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:55 smithi117 ceph-mon[32835]: pgmap v1774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:56.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:56.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:55 smithi086 ceph-mon[29229]: pgmap v1774: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:56.169 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:22:56.170 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:22:56.236 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:56.238 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:56.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:56.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:56.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:56.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:56.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:22:56.667 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:56.667 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:56.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:56.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:56.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:22:57.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:57.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:57.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:57.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:22:57.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:57.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:57.848 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:57.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:22:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:57 smithi117 ceph-mon[32835]: pgmap v1775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:57.972 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:57 smithi086 ceph-mon[29229]: pgmap v1775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:57.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:58.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:58.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:57 smithi107 ceph-mon[32381]: pgmap v1775: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:22:58.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:22:58.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:22:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:22:58.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:58.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:58.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:58.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:58.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:22:58.535 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:22:58.545 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:22:58.545 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:22:58.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:58.546 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:58.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:58.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:58.838 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:22:58.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:22:58.954 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:59.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:22:59.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:59.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:22:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:22:59.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:59.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:22:59.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:59.572 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:59.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:22:59.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:22:59.833 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:22:59.833 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:22:59.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:22:59.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:00.102 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:00.102 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:22:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:00.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:59 smithi086 ceph-mon[29229]: pgmap v1776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:00.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:59 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:00.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:00.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:22:59 smithi086 ceph-mon[29229]: osdmap e592: 12 total, 3 up, 11 in 2024-08-06T18:23:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:59 smithi117 ceph-mon[32835]: pgmap v1776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:59 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:22:59 smithi117 ceph-mon[32835]: osdmap e592: 12 total, 3 up, 11 in 2024-08-06T18:23:00.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:00.155 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:59 smithi107 ceph-mon[32381]: pgmap v1776: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:00.155 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:59 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:00.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/945009457' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:00.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:22:59 smithi107 ceph-mon[32381]: osdmap e592: 12 total, 3 up, 11 in 2024-08-06T18:23:00.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:00.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:00.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:00.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:00.534 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:00.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:00.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:00.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:01.073 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:01.108 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:01.108 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:01.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:01.241 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:01.241 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:01.242 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:01.242 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:01.242 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:01.242 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:01.242 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:01.245 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:01.247 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:01.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:01.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:01.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:01.476 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:01.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:23:01.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:01.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:01.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:02.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:02.071 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:02.071 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:02.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:02.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:01 smithi086 ceph-mon[29229]: pgmap v1778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2358011085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:01 smithi086 ceph-mon[29229]: osdmap e593: 12 total, 3 up, 11 in 2024-08-06T18:23:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:01 smithi117 ceph-mon[32835]: pgmap v1778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2358011085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:01 smithi117 ceph-mon[32835]: osdmap e593: 12 total, 3 up, 11 in 2024-08-06T18:23:02.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:02.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:01 smithi107 ceph-mon[32381]: pgmap v1778: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:02.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2358011085' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:02.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:02.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:01 smithi107 ceph-mon[32381]: osdmap e593: 12 total, 3 up, 11 in 2024-08-06T18:23:02.249 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:02.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:02.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:02.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:02.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:02.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:02.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:02.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:23:02.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:03.000 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:03.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:03.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:03.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:03.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:03.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:03.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:03.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:03.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:03.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:03.650 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:03.651 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:03.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:03.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:03 smithi107 ceph-mon[32381]: pgmap v1780: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:04.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:04.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:23:04.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:03 smithi086 ceph-mon[29229]: pgmap v1780: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:04.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:04.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:04.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:04.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:03 smithi117 ceph-mon[32835]: pgmap v1780: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:04.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:04.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:04.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:04.456 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:04.456 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:04.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:04.699 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:04.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:04.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:04.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:05.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:05.120 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:05.120 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:05.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:05.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:05.412 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:05.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:05.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:05.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:05.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:05.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:05.851 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:23:05.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:06.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:06.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:05 smithi117 ceph-mon[32835]: pgmap v1781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:06.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:05 smithi117 ceph-mon[32835]: 2.1 deep-scrub starts 2024-08-06T18:23:06.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:05 smithi117 ceph-mon[32835]: 2.1 deep-scrub ok 2024-08-06T18:23:06.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:05 smithi086 ceph-mon[29229]: pgmap v1781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:05 smithi086 ceph-mon[29229]: 2.1 deep-scrub starts 2024-08-06T18:23:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:05 smithi086 ceph-mon[29229]: 2.1 deep-scrub ok 2024-08-06T18:23:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:06.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:05 smithi107 ceph-mon[32381]: pgmap v1781: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:05 smithi107 ceph-mon[32381]: 2.1 deep-scrub starts 2024-08-06T18:23:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:05 smithi107 ceph-mon[32381]: 2.1 deep-scrub ok 2024-08-06T18:23:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:06.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:06.448 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:06.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:06.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:06.847 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:06.847 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:06.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:06 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:06 smithi117 ceph-mon[32835]: osdmap e594: 12 total, 3 up, 11 in 2024-08-06T18:23:06.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:06 smithi117 ceph-mon[32835]: osdmap e595: 12 total, 3 up, 11 in 2024-08-06T18:23:06.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:07.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:06 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:06 smithi086 ceph-mon[29229]: osdmap e594: 12 total, 3 up, 11 in 2024-08-06T18:23:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:06 smithi086 ceph-mon[29229]: osdmap e595: 12 total, 3 up, 11 in 2024-08-06T18:23:07.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:07.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:06 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:07.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1451066360' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:06 smithi107 ceph-mon[32381]: osdmap e594: 12 total, 3 up, 11 in 2024-08-06T18:23:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:06 smithi107 ceph-mon[32381]: osdmap e595: 12 total, 3 up, 11 in 2024-08-06T18:23:07.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:07.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:07.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:07.659 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:07.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:07.846 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:07.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:07 smithi107 ceph-mon[32381]: pgmap v1783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:07.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:07.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:08.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:08.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:07 smithi086 ceph-mon[29229]: pgmap v1783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:08.082 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:08.082 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:07 smithi117 ceph-mon[32835]: pgmap v1783: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:08.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:08.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:08.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:08.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:08.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:08.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:08.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:08.486 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:08.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:23:08.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:08.696 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:08.696 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:08.697 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:08.717 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:08.718 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:08.719 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:08.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:08.739 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:08.740 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:08.740 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:08.740 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:08.741 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:08.741 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:08.741 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:08.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1187369171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:08.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:08.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:09.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1187369171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:09.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:09.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1187369171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:09.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:09.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:09.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:09.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:09.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:09.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:09.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:09.844 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:09.845 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:09.892 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:09.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:09.956 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:09.956 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:09.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:09 smithi086 ceph-mon[29229]: pgmap v1785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:09.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:09.956 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:10.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:09 smithi117 ceph-mon[32835]: pgmap v1785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:10.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:09 smithi107 ceph-mon[32381]: pgmap v1785: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:10.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:10.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:10.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:10.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:10.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:10.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:10.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:10.895 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:10.895 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:11.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:11.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:11.177 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:11.177 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:11.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:23:11.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:11.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:11.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:11.578 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:23:11.578 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:23:11.578 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:23:11.578 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:23:11.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:11.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:11.838 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:11.838 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:11.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:23:11.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:11.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:11 smithi107 ceph-mon[32381]: pgmap v1786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:12.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:12.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:11 smithi086 ceph-mon[29229]: pgmap v1786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:11 smithi117 ceph-mon[32835]: pgmap v1786: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:12.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:12.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:23:12.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:12.392 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:12.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:12.517 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:12.517 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:12.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:12.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:12.911 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:13.024 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:13.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:13.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:13.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:13.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:13.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:13.578 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:13.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:13.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:13.777 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:13.844 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:13.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:14.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:13 smithi086 ceph-mon[29229]: pgmap v1787: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:14.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:13 smithi117 ceph-mon[32835]: pgmap v1787: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:14.136 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:13 smithi107 ceph-mon[32381]: pgmap v1787: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:14.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:14.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:14.408 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:14.408 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:14.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:14.651 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:14.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:14.790 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:14.790 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:14.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:15.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:15.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:14 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:14 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:15.150 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:15.165 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:15.166 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:15.166 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:15.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:15.186 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:15.187 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:15.187 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:15.187 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:15.187 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:15.202 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:15.203 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:14 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:15.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:15.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:15.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:15.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:15.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:15.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:15.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: pgmap v1788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:15.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:15.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:15.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:15.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2825460476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:15.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:15.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:15.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:16.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: pgmap v1788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:16.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2825460476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: pgmap v1788: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2825460476' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:16.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:16.238 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:16.238 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:16.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:16.266 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:16.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:16.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:16.512 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:16.513 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:16.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:16.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:16.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:16.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:16.880 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:23:16.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:17.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:17.051 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:17.051 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:16 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:23:17.051 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:16 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:23:17.051 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:17.051 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:16 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:23:17.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:16 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:23:17.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:17.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:17.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:16 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:23:17.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:16 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:23:17.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:17.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:17.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:17.585 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:17.585 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:17.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:17.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:17.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:18.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:18.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:18.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:18.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:17 smithi086 ceph-mon[29229]: pgmap v1789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:18.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:17 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:18.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:18.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:17 smithi086 ceph-mon[29229]: osdmap e596: 12 total, 3 up, 11 in 2024-08-06T18:23:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:17 smithi107 ceph-mon[32381]: pgmap v1789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:17 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:17 smithi107 ceph-mon[32381]: osdmap e596: 12 total, 3 up, 11 in 2024-08-06T18:23:18.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:18.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:23:18.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:17 smithi117 ceph-mon[32835]: pgmap v1789: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:17 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3575682250' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:17 smithi117 ceph-mon[32835]: osdmap e596: 12 total, 3 up, 11 in 2024-08-06T18:23:18.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:18.534 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:23:18.534 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:23:18.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:18.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:18.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:18.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:18.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:18.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:18.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:19.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:19.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:19.192 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:23:19.278 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:19.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:19.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:19.565 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:19.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:23:19.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:19.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:19.842 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:19.843 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:19.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:20.097 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:19 smithi086 ceph-mon[29229]: pgmap v1791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:20.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:19 smithi107 ceph-mon[32381]: pgmap v1791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:20.232 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:20.259 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:19 smithi117 ceph-mon[32835]: pgmap v1791: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:20.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:20.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:20.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:20.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:20.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:20.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:20.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:21.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:21.170 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:21.195 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:21.196 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:21.199 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:21.199 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:21.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:21.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:21.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:21.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:21.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:21.729 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:21.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:21.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:22.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:21 smithi086 ceph-mon[29229]: pgmap v1792: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:22.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:21 smithi086 ceph-mon[29229]: osdmap e597: 12 total, 3 up, 11 in 2024-08-06T18:23:22.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:22.122 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:21 smithi117 ceph-mon[32835]: pgmap v1792: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:22.122 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:21 smithi117 ceph-mon[32835]: osdmap e597: 12 total, 3 up, 11 in 2024-08-06T18:23:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:21 smithi107 ceph-mon[32381]: pgmap v1792: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:21 smithi107 ceph-mon[32381]: osdmap e597: 12 total, 3 up, 11 in 2024-08-06T18:23:22.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:22.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:22.456 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:22.456 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:22.456 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:22.472 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:22.473 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:22.473 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:22.473 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:22.473 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:22.492 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:22.493 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:22.493 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:22.493 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:22.493 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:22.493 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:22.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:22.494 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:22.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:22.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:22.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:22.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:22.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:23.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1833220207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:22 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:23.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:23.172 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1833220207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:23.172 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:22 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:23.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:23.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1833220207' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:23.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:22 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:23.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:23.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:23.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:23.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:23.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:24.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:24.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:23 smithi086 ceph-mon[29229]: pgmap v1794: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:23 smithi117 ceph-mon[32835]: pgmap v1794: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:23 smithi107 ceph-mon[32381]: pgmap v1794: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:24.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:24.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:24.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:24.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:24.774 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:24.774 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:24.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:24.894 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:23:25.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:25.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:25.172 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:25.172 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:25.172 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.244 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:457,from:0,reservation_nonce:0)", 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 586.65595288099996, 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00021816399999999999, 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.245 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353002+0000", 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353003+0000", 2024-08-06T18:23:25.246 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8599999999999998e-07 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353006+0000", 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0249999999999998e-06 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353015+0000", 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9740000000000008e-06 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.247 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353020+0000", 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2449999999999998e-06 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353061+0000", 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1219e-05 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353079+0000", 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7915999999999999e-05 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.248 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353220+0000", 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000141099 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 586.655246121, 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6941000000000007e-05, 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.249 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.250 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353709+0000", 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353710+0000", 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8500000000000001e-07 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.251 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353714+0000", 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8810000000000004e-06 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353724+0000", 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1630000000000002e-06 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353728+0000", 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3309999999999997e-06 2024-08-06T18:23:25.252 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353769+0000", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0890000000000003e-05 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353777+0000", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9880000000000004e-06 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:13:38.353796+0000", 2024-08-06T18:23:25.253 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9003000000000001e-05 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:476,from:0,reservation_nonce:0)", 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 520.58770516300001, 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018068000000000001, 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.254 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421250+0000", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421251+0000", 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8000000000000005e-07 2024-08-06T18:23:25.255 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421254+0000", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9890000000000001e-06 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421263+0000", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3430000000000002e-06 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421269+0000", 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8270000000000003e-06 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.256 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421316+0000", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7132000000000002e-05 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421328+0000", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2038999999999999e-05 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421431+0000", 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010257 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.257 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 520.58703320699999, 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4305e-05, 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.258 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:23:25.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421922+0000", 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.13e-07 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421926+0000", 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4400000000000001e-06 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421934+0000", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6200000000000005e-06 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421941+0000", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6229999999999999e-06 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421977+0000", 2024-08-06T18:23:25.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5834999999999998e-05 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.421986+0000", 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1279999999999997e-06 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:14:44.422006+0000", 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0046000000000001e-05 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:509,from:0,reservation_nonce:0)", 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 367.84675656399997, 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018985599999999999, 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162198+0000", 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162200+0000", 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3599999999999999e-06 2024-08-06T18:23:25.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162205+0000", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.938e-06 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162219+0000", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3974e-05 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162227+0000", 2024-08-06T18:23:25.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-06 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162279+0000", 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2927999999999999e-05 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162293+0000", 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3737000000000001e-05 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.266 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162388+0000", 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.5088999999999997e-05 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 367.84611548599997, 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.3175000000000003e-05, 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.267 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:23:25.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162840+0000", 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1900000000000002e-07 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162843+0000", 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3230000000000002e-06 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162852+0000", 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5979999999999997e-06 2024-08-06T18:23:25.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162858+0000", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.19e-06 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162889+0000", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1059000000000002e-05 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162896+0000", 2024-08-06T18:23:25.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.635e-06 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162913+0000", 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6750999999999999e-05 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:23:25.272 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 182.62114887800001, 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017163700000000001, 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387807+0000", 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0500000000000002e-07 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387810+0000", 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3239999999999999e-06 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387819+0000", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387824+0000", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3630000000000002e-06 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387861+0000", 2024-08-06T18:23:25.275 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6433000000000002e-05 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387875+0000", 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4549000000000001e-05 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387978+0000", 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010249 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.276 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 182.620509492, 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2353000000000003e-05, 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.277 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4600000000000005e-07 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388449+0000", 2024-08-06T18:23:25.278 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.174e-06 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388457+0000", 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1240000000000005e-06 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388462+0000", 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4399999999999998e-06 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:23:25.279 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388493+0000", 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1256e-05 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388500+0000", 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4690000000000002e-06 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388518+0000", 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8343999999999999e-05 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.280 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:23:25.281 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:23:25.281 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:23:25.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:25.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:25.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:25.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:25.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:25.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:23:25.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:25.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:25.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:25.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:25 smithi086 ceph-mon[29229]: pgmap v1795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:25.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:25 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:25.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:25.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:25 smithi086 ceph-mon[29229]: osdmap e598: 12 total, 3 up, 11 in 2024-08-06T18:23:26.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:26.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:25 smithi117 ceph-mon[32835]: pgmap v1795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:26.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:25 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:26.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:26.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:25 smithi117 ceph-mon[32835]: osdmap e598: 12 total, 3 up, 11 in 2024-08-06T18:23:26.158 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:26.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:25 smithi107 ceph-mon[32381]: pgmap v1795: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:25 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3553086134' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:25 smithi107 ceph-mon[32381]: osdmap e598: 12 total, 3 up, 11 in 2024-08-06T18:23:26.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:26.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:26.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:26.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:26.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:26.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:26.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:26.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:26.748 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:26.749 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:26.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:23:26.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:26.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:27.043 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:27.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:27.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:27.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:27.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:27.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:27.407 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:27.407 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:27.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:27.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:27.748 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:27.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:27.811 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:27 smithi117 ceph-mon[32835]: pgmap v1797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:27.811 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:27 smithi117 ceph-mon[32835]: osdmap e599: 12 total, 3 up, 11 in 2024-08-06T18:23:27.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:27 smithi086 ceph-mon[29229]: pgmap v1797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:27.857 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:27 smithi086 ceph-mon[29229]: osdmap e599: 12 total, 3 up, 11 in 2024-08-06T18:23:27.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:27 smithi107 ceph-mon[32381]: pgmap v1797: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:27 smithi107 ceph-mon[32381]: osdmap e599: 12 total, 3 up, 11 in 2024-08-06T18:23:28.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:28.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:28.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:28.344 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:28.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:23:28.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:23:28.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:28.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:28.602 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:28.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:28.789 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:28.790 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:28.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:28 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:23:28.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:28 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:23:28.809 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:28.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:28.829 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:28.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:28.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:28.832 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:28.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:28.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:28 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:23:28.870 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:28 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:23:28.871 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:28.871 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:28 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:23:28.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:28 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:23:28.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:29.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:29.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:29.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:29.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:29.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:29.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:29.801 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:29.801 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:29 smithi117 ceph-mon[32835]: pgmap v1799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2616893060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:29.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:29 smithi107 ceph-mon[32381]: pgmap v1799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:29.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2616893060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:29.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.997 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:29.998 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:29 smithi086 ceph-mon[29229]: pgmap v1799: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2616893060' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:29.998 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:30.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:30.170 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:30.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:30.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:30.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:30.342 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:30.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:30.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:30.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:30.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:23:30.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:30.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:31.066 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:31.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:31.187 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:31.188 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:31.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:31.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:31.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:31.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:31.706 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:23:31.785 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:31 smithi107 ceph-mon[32381]: pgmap v1800: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:31.786 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:31.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:31.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:23:31.918 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:32.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:31 smithi086 ceph-mon[29229]: pgmap v1800: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:32.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:32.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:32.073 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:31 smithi117 ceph-mon[32835]: pgmap v1800: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:32.073 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:32.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:32.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:32.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:32.547 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:32.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:32.584 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:32.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:32.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:32 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:32.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:32 smithi117 ceph-mon[32835]: osdmap e600: 12 total, 3 up, 11 in 2024-08-06T18:23:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:32 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:32 smithi107 ceph-mon[32381]: osdmap e600: 12 total, 3 up, 11 in 2024-08-06T18:23:32.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:32 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3428561629' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:32 smithi086 ceph-mon[29229]: osdmap e600: 12 total, 3 up, 11 in 2024-08-06T18:23:33.093 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:33.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:23:33.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:33.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:33.343 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:33.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:33.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:33.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:33.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:33.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:33.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:33.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:33 smithi107 ceph-mon[32381]: pgmap v1802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:33.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:33 smithi086 ceph-mon[29229]: pgmap v1802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:34.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:33 smithi117 ceph-mon[32835]: pgmap v1802: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:34.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:34.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:34.393 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:34.393 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:34.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:34.410 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:34.410 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:34.411 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:34.411 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:34.411 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:34.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:34.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:34.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:34.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:23:34.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:34.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:34.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:34.893 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/34965706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:34.937 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:34.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:35.018 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/34965706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:35.018 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:35.018 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:35.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:35.095 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/34965706' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:35.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:35.157 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:35.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:35.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:35.525 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:35.540 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:35.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:23:35.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:35.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:35.791 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:35.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:35 smithi086 ceph-mon[29229]: pgmap v1803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:35.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:36.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:36.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:35 smithi117 ceph-mon[32835]: pgmap v1803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:36.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.081 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:35 smithi107 ceph-mon[32381]: pgmap v1803: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:36.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:36.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:36.304 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:36.304 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:36.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:36.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:36.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:36.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:36.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:36.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:37.034 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:23:37.035+0000 7fa521335640 0 --2- 172.21.15.86:0/1477498320 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fa51c107c30 0x7fa51c1a0010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:23:37.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:23:37.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:37.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:37.330 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:37.345 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:37.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:37.383 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:37.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:37.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:37.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:37.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:37.729 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:37.729 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:37.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:37 smithi086 ceph-mon[29229]: pgmap v1804: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:37.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:37 smithi086 ceph-mon[29229]: osdmap e601: 12 total, 3 up, 11 in 2024-08-06T18:23:37.730 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:37.767 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:23:37.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:37.884 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:37 smithi107 ceph-mon[32381]: pgmap v1804: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:37.884 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:37 smithi107 ceph-mon[32381]: osdmap e601: 12 total, 3 up, 11 in 2024-08-06T18:23:37.884 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:37.911 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:37.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:23:37.915 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:38.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:38.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:37 smithi117 ceph-mon[32835]: pgmap v1804: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:38.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:37 smithi117 ceph-mon[32835]: osdmap e601: 12 total, 3 up, 11 in 2024-08-06T18:23:38.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:23:38.163 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:38.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:38.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:38.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:38.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:38.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:38.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:38 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:38.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:38.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:38 smithi086 ceph-mon[29229]: osdmap e602: 12 total, 3 up, 11 in 2024-08-06T18:23:38.840 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:38 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:38.841 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:38.841 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:38 smithi107 ceph-mon[32381]: osdmap e602: 12 total, 3 up, 11 in 2024-08-06T18:23:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:38 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1477498320' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:23:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:38 smithi117 ceph-mon[32835]: osdmap e602: 12 total, 3 up, 11 in 2024-08-06T18:23:38.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:39.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:39.140 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:39.140 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:39.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:39.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:39.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:39.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:39.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:39.573 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:39.573 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:39.591 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:39.592 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:39.610 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:39.611 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:39.660 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:39.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:39.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:39.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:39 smithi086 ceph-mon[29229]: pgmap v1807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:39.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2963811654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:39.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:39.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:39.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:39 smithi107 ceph-mon[32381]: pgmap v1807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:39.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2963811654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:40.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:39 smithi117 ceph-mon[32835]: pgmap v1807: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2963811654' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:40.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:40.247 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:40.247 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:40.273 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:40.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:23:40.313 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:40.314 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:40.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:40.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:40.510 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:40.510 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:40.526 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:40.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:40.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:23:40.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:40.766 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:40.766 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:40.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:40.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:40.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:40.951 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:40.992 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:41.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:41.041 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.041 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.041 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:41.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:41.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:41.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:41.545 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:41.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:41.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:41.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:41.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:41.808 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:41 smithi107 ceph-mon[32381]: pgmap v1808: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:41.809 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:41 smithi107 ceph-mon[32381]: osdmap e603: 12 total, 3 up, 11 in 2024-08-06T18:23:41.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:42.016 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:23:42.016 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:23:42.074 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:42.077 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:42.077 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:42.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:41 smithi086 ceph-mon[29229]: pgmap v1808: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:42.077 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:41 smithi086 ceph-mon[29229]: osdmap e603: 12 total, 3 up, 11 in 2024-08-06T18:23:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:41 smithi117 ceph-mon[32835]: pgmap v1808: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:41 smithi117 ceph-mon[32835]: osdmap e603: 12 total, 3 up, 11 in 2024-08-06T18:23:42.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:42.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:23:42.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:42.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:42.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:42.628 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:42.628 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:42.692 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:42.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:23:42.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:42.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:42.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:42.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:42.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:42.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:42.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:42.967 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:43.024 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:43.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:43.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:43.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:23:43.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:23:43.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:43.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:43.348 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:23:43.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:43.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:43.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:43.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:43.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:43.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:23:43.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:23:43.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:43.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:43.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:43.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:43 smithi086 ceph-mon[29229]: osdmap e604: 12 total, 3 up, 11 in 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:43.986 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:43 smithi107 ceph-mon[32381]: osdmap e604: 12 total, 3 up, 11 in 2024-08-06T18:23:44.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/135317290' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:43 smithi117 ceph-mon[32835]: osdmap e604: 12 total, 3 up, 11 in 2024-08-06T18:23:44.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:44.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:44.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:44.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:44.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:44.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:44.463 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:23:44.464 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:23:44.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:44.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:44.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:44.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:45.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:45.036 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:45.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:23:45.051 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:45.051 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:45.052 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:45.053 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:45.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:45.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:45.308 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:45.308 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:45.338 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:45.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:45.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:45.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:45.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:45.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:45.768 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:45.768 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:45.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:45 smithi117 ceph-mon[32835]: pgmap v1812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/661679197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:45.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:45.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:45 smithi107 ceph-mon[32381]: pgmap v1812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/661679197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:45.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:23:46.018 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:45 smithi086 ceph-mon[29229]: pgmap v1812: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:46.018 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/661679197' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:46.018 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:46.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:46.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:46.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:46.244 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:46.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:46.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:46.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:46.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:23:46.863 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.45914697399999, 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000266113, 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.864 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404599+0000", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.023e-06 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404602+0000", 2024-08-06T18:23:46.865 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.534e-06 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404616+0000", 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3881e-05 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404626+0000", 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0402e-05 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.866 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404697+0000", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0785999999999996e-05 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404716+0000", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9258000000000001e-05 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404864+0000", 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147229 2024-08-06T18:23:46.867 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 201.45837239400001, 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014078100000000001, 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405373+0000", 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4000000000000001e-07 2024-08-06T18:23:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405377+0000", 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.391e-06 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405387+0000", 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0040999999999999e-05 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405393+0000", 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.976e-06 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405481+0000", 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7995e-05 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405490+0000", 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1649999999999995e-06 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.871 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405513+0000", 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2572999999999999e-05 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:575,from:0,reservation_nonce:0)", 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 119.28572349300001, 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039962, 2024-08-06T18:23:46.872 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.873 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578023+0000", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.736e-06 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578030+0000", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0140000000000001e-06 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578056+0000", 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6302999999999999e-05 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.874 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578093+0000", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6529000000000002e-05 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578174+0000", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1209000000000001e-05 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578203+0000", 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9045999999999999e-05 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.875 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578421+0000", 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021778300000000001 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 119.284309529, 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030426500000000002, 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.876 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.877 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579437+0000", 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.734e-06 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579490+0000", 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3355e-05 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.878 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579515+0000", 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5236999999999999e-05 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579527+0000", 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1482e-05 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579637+0000", 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011032200000000001 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.879 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579666+0000", 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9186000000000001e-05 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579739+0000", 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2948999999999996e-05 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.880 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:599,from:0,reservation_nonce:0)", 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.081374393000001, 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188605, 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.881 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782371+0000", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8299999999999995e-07 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782375+0000", 2024-08-06T18:23:46.882 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7780000000000001e-06 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782384+0000", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.986e-06 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782392+0000", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6650000000000003e-06 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782430+0000", 2024-08-06T18:23:46.883 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8778e-05 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782444+0000", 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3981999999999999e-05 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782559+0000", 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011443299999999999 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.884 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 19.080726112000001, 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5184000000000005e-05, 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:23:46.885 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783019+0000", 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7199999999999999e-07 2024-08-06T18:23:46.886 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783022+0000", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4340000000000001e-06 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783029+0000", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6590000000000001e-06 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783032+0000", 2024-08-06T18:23:46.887 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2650000000000001e-06 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783061+0000", 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8751999999999998e-05 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783067+0000", 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:23:46.888 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783084+0000", 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:23:46.889 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:23:46.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:46.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:46.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:46 smithi107 ceph-mon[32381]: osdmap e605: 12 total, 3 up, 11 in 2024-08-06T18:23:47.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:47.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:47.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:46 smithi117 ceph-mon[32835]: osdmap e605: 12 total, 3 up, 11 in 2024-08-06T18:23:47.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:47.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:46 smithi086 ceph-mon[29229]: osdmap e605: 12 total, 3 up, 11 in 2024-08-06T18:23:47.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:47.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:47.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:47.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:23:47.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:47.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:47.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:23:47.775 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:47.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:47.909 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:47 smithi086 ceph-mon[29229]: pgmap v1813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:47.910 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:47.910 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:47.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:48.012 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:47 smithi107 ceph-mon[32381]: pgmap v1813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:48.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:48.108 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:47 smithi117 ceph-mon[32835]: pgmap v1813: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:48.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:48.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:48.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:48.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:48.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:48.800 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:23:48.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:48.909 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:49.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:49.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:23:49.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:49.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:49.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:49.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:49.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:49.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:49.389 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:49.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:49.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:49.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:49.737 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:49.737 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:49.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:49.909 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:49.909 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: pgmap v1815: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:49.910 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:49 smithi107 ceph-mon[32381]: osdmap e606: 12 total, 3 up, 11 in 2024-08-06T18:23:49.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:49.987 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:49.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:23:49.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:49.996 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:49.997 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:49.997 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: pgmap v1815: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:49.997 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:49 smithi086 ceph-mon[29229]: osdmap e606: 12 total, 3 up, 11 in 2024-08-06T18:23:50.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: pgmap v1815: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2834264717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:23:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:49 smithi117 ceph-mon[32835]: osdmap e606: 12 total, 3 up, 11 in 2024-08-06T18:23:50.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:50.240 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:50.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:50.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:50.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:50.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:50.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:50.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:50.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:50.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:50.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:50.802 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:50.802 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:50.802 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:50.803 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:50.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:50.805 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:50.805 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:50.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:50.822 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:50.822 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:50.823 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:50.823 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:50.823 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:50.823 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:50.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:50.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/882833047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/882833047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:51.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:51.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/882833047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:51.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:51.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:51.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:51.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:51.589 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:51.589 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:51.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:23:51.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:51.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:51.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:51.753 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:51.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:51.870 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:51.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:51.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:51.993 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:51 smithi086 ceph-mon[29229]: pgmap v1817: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:51.993 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:51 smithi086 ceph-mon[29229]: osdmap e607: 12 total, 3 up, 11 in 2024-08-06T18:23:52.069 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:52.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:51 smithi117 ceph-mon[32835]: pgmap v1817: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:52.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:51 smithi117 ceph-mon[32835]: osdmap e607: 12 total, 3 up, 11 in 2024-08-06T18:23:52.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:51 smithi107 ceph-mon[32381]: pgmap v1817: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 328 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:51 smithi107 ceph-mon[32381]: osdmap e607: 12 total, 3 up, 11 in 2024-08-06T18:23:52.275 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:52.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:23:52.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:52.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:52.520 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:23:52.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:52.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:52.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:52.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:52.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:53.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:53.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:53.170 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:53.170 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:53.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:53.293 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:53.293 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:53.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:53.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:53.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:53.569 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:53.590 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:23:53.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:53.807 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:23:53.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:53.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:53 smithi086 ceph-mon[29229]: pgmap v1819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:53.821 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:53.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:53.949 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:53.950 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:53.950 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:53.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:53 smithi107 ceph-mon[32381]: pgmap v1819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:53.950 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:54.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:53 smithi117 ceph-mon[32835]: pgmap v1819: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:23:54.203 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:54.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:54.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:54.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:54.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:54.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:54.626 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:54.632 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:54.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:23:54.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:54.816 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:54.816 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:54.816 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:54.816 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:54.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:54.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:54.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:54.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:54 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:54.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:54 smithi086 ceph-mon[29229]: osdmap e608: 12 total, 3 up, 11 in 2024-08-06T18:23:54.888 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:23:54.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:55.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:54 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:55.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:55.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:54 smithi117 ceph-mon[32835]: osdmap e608: 12 total, 3 up, 11 in 2024-08-06T18:23:55.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:55.143 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:55.144 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:54 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:23:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4176972320' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:23:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:54 smithi107 ceph-mon[32381]: osdmap e608: 12 total, 3 up, 11 in 2024-08-06T18:23:55.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:55.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:55.433 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:55.434 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:55.434 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:55.434 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:55.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:55.647 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:55.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:55.692 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:55.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:55.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:55.933 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:55 smithi117 ceph-mon[32835]: pgmap v1821: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:55.968 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:55 smithi086 ceph-mon[29229]: pgmap v1821: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:56.042 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:56.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:23:56.176 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:23:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:23:56.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:23:56.193 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:23:56.193 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:23:56.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:23:56.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:23:56.194 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:23:56.211 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:23:56.211 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:23:56.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:23:56.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:23:56.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:23:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:55 smithi107 ceph-mon[32381]: pgmap v1821: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:56.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:23:56.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:23:56.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:56.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:56.509 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:56.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:56.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:23:56.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:56.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:23:56.901 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:23:56.959 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:56.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2981556426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:56 smithi086 ceph-mon[29229]: osdmap e609: 12 total, 3 up, 11 in 2024-08-06T18:23:57.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2981556426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:57.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:56 smithi117 ceph-mon[32835]: osdmap e609: 12 total, 3 up, 11 in 2024-08-06T18:23:57.098 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:57.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:23:57.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:23:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2981556426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:23:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:57.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:56 smithi107 ceph-mon[32381]: osdmap e609: 12 total, 3 up, 11 in 2024-08-06T18:23:57.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:57.351 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:23:57.352 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:23:57.352 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:23:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:23:57.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:23:57.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:57.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:57.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:57.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:23:58.030 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:57 smithi086 ceph-mon[29229]: pgmap v1822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:57 smithi117 ceph-mon[32835]: pgmap v1822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:58.164 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:58.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:23:58.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:57 smithi107 ceph-mon[32381]: pgmap v1822: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:58.189 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:58.189 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:58.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:23:58.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:23:58.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:23:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:23:58.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:58.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:58.781 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:23:58.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:23:58.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:23:58.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:59.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:59.154 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:23:59.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:23:59.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:59.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:23:59.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:23:59.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:23:59.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:23:59.581 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:23:59.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:23:59.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:23:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:23:59.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:23:59.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:23:59.830 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:23:59.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:23:59.935 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:59 smithi086 ceph-mon[29229]: pgmap v1824: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:23:59.936 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:23:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:00.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:00.078 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:59 smithi107 ceph-mon[32381]: pgmap v1824: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:00.078 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:23:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:59 smithi117 ceph-mon[32835]: pgmap v1824: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:23:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:00.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:00.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:00.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:00.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:00.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:00.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:00.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:00.833 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:00.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:00 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:00.854 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:00.855 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:00 smithi117 ceph-mon[32835]: osdmap e610: 12 total, 3 up, 11 in 2024-08-06T18:24:00.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:01.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:00 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:01.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:01.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:00 smithi086 ceph-mon[29229]: osdmap e610: 12 total, 3 up, 11 in 2024-08-06T18:24:01.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:01.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:01.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:01.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:00 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:01.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3436112452' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:01.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:00 smithi107 ceph-mon[32381]: osdmap e610: 12 total, 3 up, 11 in 2024-08-06T18:24:01.125 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:01.125 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:01.134 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:01.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:01.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:01.392 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:01.392 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:01.497 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:01.498 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:01.498 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:01.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:01.514 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:01.515 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:01.515 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:01.516 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:01.516 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:01.532 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:01.533 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:01.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:01.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:01.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:01.534 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:01.534 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:01.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:01.765 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:01.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:01.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:24:01.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:02.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:02.077 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:01 smithi086 ceph-mon[29229]: pgmap v1826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2673778935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:01 smithi086 ceph-mon[29229]: osdmap e611: 12 total, 3 up, 11 in 2024-08-06T18:24:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:01 smithi117 ceph-mon[32835]: pgmap v1826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2673778935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:01 smithi117 ceph-mon[32835]: osdmap e611: 12 total, 3 up, 11 in 2024-08-06T18:24:02.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:02.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:01 smithi107 ceph-mon[32381]: pgmap v1826: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:02.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2673778935' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:02.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:01 smithi107 ceph-mon[32381]: osdmap e611: 12 total, 3 up, 11 in 2024-08-06T18:24:02.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:02.194 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:02.219 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:02.219 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:02.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:02.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:02.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:02.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:02.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:02.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:02.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:02.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:02.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:02.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:03.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:24:03.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:03.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:03.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:03.304 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:03.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:03.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:03.590 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:03.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:03.652 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:03.652 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:03.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:03.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:04.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:04.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:03 smithi086 ceph-mon[29229]: pgmap v1828: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:04.047 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:04.047 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:03 smithi117 ceph-mon[32835]: pgmap v1828: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:04.151 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:04.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:24:04.156 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:04.181 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:03 smithi107 ceph-mon[32381]: pgmap v1828: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:04.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:24:04.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:04.396 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:04.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:04.533 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:04.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:04.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:04.707 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:04.708 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:04.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:04.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:05.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:05.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:05.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:05.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:05.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:05.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:05.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:05.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:05.844 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:24:05.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:05 smithi117 ceph-mon[32835]: pgmap v1829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:05.887 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:05 smithi107 ceph-mon[32381]: pgmap v1829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:05.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:05.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:06.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:05 smithi086 ceph-mon[29229]: pgmap v1829: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:06.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:06.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:06.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:06.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:24:06.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:06.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:06.722 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:06.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:06.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:07.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:07.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:06 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:07.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:06 smithi086 ceph-mon[29229]: osdmap e612: 12 total, 3 up, 11 in 2024-08-06T18:24:07.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:06 smithi086 ceph-mon[29229]: osdmap e613: 12 total, 3 up, 11 in 2024-08-06T18:24:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:06 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:06 smithi117 ceph-mon[32835]: osdmap e612: 12 total, 3 up, 11 in 2024-08-06T18:24:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:06 smithi117 ceph-mon[32835]: osdmap e613: 12 total, 3 up, 11 in 2024-08-06T18:24:07.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:07.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:07.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:06 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3998799178' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:06 smithi107 ceph-mon[32381]: osdmap e612: 12 total, 3 up, 11 in 2024-08-06T18:24:07.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:06 smithi107 ceph-mon[32381]: osdmap e613: 12 total, 3 up, 11 in 2024-08-06T18:24:07.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:07.400 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:07.401 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:07.401 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:07.417 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:07.418 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:07.435 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:07.436 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:07.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:07.468 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:07.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:07.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:07.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:07.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:07 smithi107 ceph-mon[32381]: pgmap v1831: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2377428054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:07.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:08.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:07 smithi117 ceph-mon[32835]: pgmap v1831: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:08.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2377428054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:08.069 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:08.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:08.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:07 smithi086 ceph-mon[29229]: pgmap v1831: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2377428054' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:08.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:08.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:08.202 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:08.203 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:08.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:08.273 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:08.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:08.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:08.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:08.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:08.821 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:08.821 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:08.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:24:08.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:08.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:24:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:09.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:09.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:09.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:09.121 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:09.191 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:09.194 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:09.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:09.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:09.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:09.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:09.528 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:09.529 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:09.529 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:09.529 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:09.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:09.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:09.751 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:09.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:09.861 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:09.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: pgmap v1833: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=239571350,config=507150784,health=3284250782,kv=1412558022}) 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=2672015887}) 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257241001}) 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110795328}) 2024-08-06T18:24:10.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3505908715}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2725406177}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=332831878}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T18:24:10.022 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T18:24:10.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T18:24:10.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T18:24:10.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T18:24:10.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T18:24:10.023 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35217734}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742585113}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692546528}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733765210}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412170737}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672818670}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515870135}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152575532}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707336588}) 2024-08-06T18:24:10.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376818176}) 2024-08-06T18:24:10.025 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923567500}) 2024-08-06T18:24:10.025 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3227500486}) 2024-08-06T18:24:10.025 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651641586}) 2024-08-06T18:24:10.025 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324850269}) 2024-08-06T18:24:10.025 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586288064}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032585628}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1722138047}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761684411}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759762289}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916015270}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749132440}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689297570}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082421406}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4034547109}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306514798}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237291990}) 2024-08-06T18:24:10.026 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881318396}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640983107}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500614668}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942862584}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707082902}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277380450}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362239312}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954139931}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428969802}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948140968}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628285778}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756828794}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440114928}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846226870}) 2024-08-06T18:24:10.027 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140632320}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132469168}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=108355425}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351934558}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195547408}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750979438}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3620550501}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886732918}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1382514422}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=4} crc {logm=2726720850,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=8248873,mon_config_key=1409348696}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1812017197}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3827339159}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=3253882910,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=997904580,osdmap=525774843}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1194055980}) 2024-08-06T18:24:10.028 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=62377387}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3950393464}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=756468757}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=594924948}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1724773527}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2286344386}) 2024-08-06T18:24:10.029 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3508162101}) 2024-08-06T18:24:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: pgmap v1833: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=239571350,config=507150784,health=3284250782,kv=1412558022}) 2024-08-06T18:24:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=2672015887}) 2024-08-06T18:24:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257241001}) 2024-08-06T18:24:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110795328}) 2024-08-06T18:24:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3505908715}) 2024-08-06T18:24:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2725406177}) 2024-08-06T18:24:10.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=332831878}) 2024-08-06T18:24:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T18:24:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T18:24:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T18:24:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T18:24:10.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35217734}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742585113}) 2024-08-06T18:24:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692546528}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733765210}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412170737}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672818670}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515870135}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152575532}) 2024-08-06T18:24:10.128 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707336588}) 2024-08-06T18:24:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376818176}) 2024-08-06T18:24:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923567500}) 2024-08-06T18:24:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3227500486}) 2024-08-06T18:24:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651641586}) 2024-08-06T18:24:10.129 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324850269}) 2024-08-06T18:24:10.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586288064}) 2024-08-06T18:24:10.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032585628}) 2024-08-06T18:24:10.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1722138047}) 2024-08-06T18:24:10.130 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761684411}) 2024-08-06T18:24:10.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:10.132 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759762289}) 2024-08-06T18:24:10.132 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916015270}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749132440}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689297570}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082421406}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4034547109}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306514798}) 2024-08-06T18:24:10.133 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237291990}) 2024-08-06T18:24:10.134 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881318396}) 2024-08-06T18:24:10.134 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640983107}) 2024-08-06T18:24:10.134 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500614668}) 2024-08-06T18:24:10.134 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942862584}) 2024-08-06T18:24:10.134 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707082902}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277380450}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362239312}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954139931}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428969802}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948140968}) 2024-08-06T18:24:10.135 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628285778}) 2024-08-06T18:24:10.136 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756828794}) 2024-08-06T18:24:10.136 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440114928}) 2024-08-06T18:24:10.136 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846226870}) 2024-08-06T18:24:10.136 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140632320}) 2024-08-06T18:24:10.136 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132469168}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=108355425}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351934558}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195547408}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750979438}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3620550501}) 2024-08-06T18:24:10.137 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886732918}) 2024-08-06T18:24:10.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1382514422}) 2024-08-06T18:24:10.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=4} crc {logm=2726720850,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=8248873,mon_config_key=1409348696}) 2024-08-06T18:24:10.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1812017197}) 2024-08-06T18:24:10.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3827339159}) 2024-08-06T18:24:10.138 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=3253882910,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=997904580,osdmap=525774843}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1194055980}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=62377387}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3950393464}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=756468757}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=594924948}) 2024-08-06T18:24:10.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1724773527}) 2024-08-06T18:24:10.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:24:10.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2286344386}) 2024-08-06T18:24:10.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3508162101}) 2024-08-06T18:24:10.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:24:10.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: pgmap v1833: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:10.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=26,config=2,health=12,kv=60} crc {auth=239571350,config=507150784,health=3284250782,kv=1412558022}) 2024-08-06T18:24:10.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {kv=1,logm=99} crc {kv=120296119,logm=2672015887}) 2024-08-06T18:24:10.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3257241001}) 2024-08-06T18:24:10.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2110795328}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3505908715}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2725406177}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=332831878}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749161675}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=281849401}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1135472887}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=829308359}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2649094070}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=657453312}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3032255625}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=820927254}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=35217734}) 2024-08-06T18:24:10.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3742585113}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=692546528}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1733765210}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1412170737}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3672818670}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1515870135}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3152575532}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2707336588}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=376818176}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3923567500}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3227500486}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2651641586}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1324850269}) 2024-08-06T18:24:10.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=586288064}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2032585628}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1722138047}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=761684411}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2759762289}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3916015270}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3749132440}) 2024-08-06T18:24:10.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1689297570}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3082421406}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4034547109}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2306514798}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2237291990}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=881318396}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2640983107}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2500614668}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1942862584}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3707082902}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2277380450}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2362239312}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3954139931}) 2024-08-06T18:24:10.232 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428969802}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1948140968}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=628285778}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756828794}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440114928}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2846226870}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2140632320}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1132469168}) 2024-08-06T18:24:10.233 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=108355425}) 2024-08-06T18:24:10.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2351934558}) 2024-08-06T18:24:10.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2195547408}) 2024-08-06T18:24:10.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=750979438}) 2024-08-06T18:24:10.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3620550501}) 2024-08-06T18:24:10.234 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3886732918}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1382514422}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=55,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=4} crc {logm=2726720850,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=8248873,mon_config_key=1409348696}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1812017197}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3827339159}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=75,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=7} crc {mon_config_key=3253882910,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=997904580,osdmap=525774843}) 2024-08-06T18:24:10.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1194055980}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=62377387}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3950393464}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=756468757}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=594924948}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1724773527}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2286344386}) 2024-08-06T18:24:10.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=78} crc {osdmap=3508162101}) 2024-08-06T18:24:10.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:10.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:10.482 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:10.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:10.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:10.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:10.621 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:10.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:10.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:10.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:11.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:11.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:11.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:11.350 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:11.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:24:11.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:11.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:11.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:11.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:11.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:11.595 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:11.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:11.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:11.868 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:24:11.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:11.985 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:11.985 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:11.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:11 smithi086 ceph-mon[29229]: pgmap v1834: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:11.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:12.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:11 smithi117 ceph-mon[32835]: pgmap v1834: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:12.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:12.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:12.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:12.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:11 smithi107 ceph-mon[32381]: pgmap v1834: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:12.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:12.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:12.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:12.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:12.298 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:12.298 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:12.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:12.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:12.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:12.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:12.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:13.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:12 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:13.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:13.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:12 smithi086 ceph-mon[29229]: osdmap e614: 12 total, 3 up, 11 in 2024-08-06T18:24:13.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:13.117 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:13.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:12 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:12 smithi107 ceph-mon[32381]: osdmap e614: 12 total, 3 up, 11 in 2024-08-06T18:24:13.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:13.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:13.343 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:13.366 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:12 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:13.366 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4032746889' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:13.367 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:12 smithi117 ceph-mon[32835]: osdmap e614: 12 total, 3 up, 11 in 2024-08-06T18:24:13.376 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:13.376 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:13.393 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:13.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:13.411 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:13.412 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:13.412 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:13.412 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:13.412 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:13.413 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:13.413 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:13.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:13.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:13.713 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:13.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:24:13.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:13.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:13.961 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:14.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:14.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:13 smithi086 ceph-mon[29229]: pgmap v1836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:14.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3924875368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:14.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:14.126 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:14.126 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:14.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:14.174 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:13 smithi117 ceph-mon[32835]: pgmap v1836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:14.174 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3924875368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:14.174 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:13 smithi107 ceph-mon[32381]: pgmap v1836: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3924875368' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:14.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:14.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:14.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:14.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:14.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:24:14.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:14.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:14.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:14.970 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:14.971 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:15.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:15.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:15.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:15.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:15.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:15.471 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:15.476 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:15.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:15.634 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:15.634 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:15.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:15.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:16.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:16.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:15 smithi117 ceph-mon[32835]: pgmap v1837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:16.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:24:16.136 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:16.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:24:16.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:16.149 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:15 smithi086 ceph-mon[29229]: pgmap v1837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:16.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:15 smithi107 ceph-mon[32381]: pgmap v1837: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:16.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:16.384 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:16.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:16.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:16.602 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:16.602 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:16.676 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:16.711 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:16.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:16.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:16.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:16.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:17.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:17.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:17.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:17.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:17.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:17.720 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:24:17.732 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:17 smithi086 ceph-mon[29229]: pgmap v1838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:17.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:17 smithi086 ceph-mon[29229]: osdmap e615: 12 total, 3 up, 11 in 2024-08-06T18:24:17.733 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:17.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:17.892 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:17.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:18.004 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:18.004 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:18.004 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:17 smithi107 ceph-mon[32381]: pgmap v1838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:18.004 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:17 smithi107 ceph-mon[32381]: osdmap e615: 12 total, 3 up, 11 in 2024-08-06T18:24:18.004 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:18.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:17 smithi117 ceph-mon[32835]: pgmap v1838: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:17 smithi117 ceph-mon[32835]: osdmap e615: 12 total, 3 up, 11 in 2024-08-06T18:24:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:18.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:18.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:18.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:24:18.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:18.613 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:18.613 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:18.614 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:18.630 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:18.631 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:18.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:18.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:18.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:18.648 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:18.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:18.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:18.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:18.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:18.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:18.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:18.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:18.718 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:18.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:18.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:18 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:18.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:18.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:18 smithi086 ceph-mon[29229]: osdmap e616: 12 total, 3 up, 11 in 2024-08-06T18:24:18.803 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/778147124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:18.829 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:18 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:18.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:18.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:18 smithi117 ceph-mon[32835]: osdmap e616: 12 total, 3 up, 11 in 2024-08-06T18:24:18.830 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/778147124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:18.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:19.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:19.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:19.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:19.183 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:18 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:19.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1264781597' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:19.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:18 smithi107 ceph-mon[32381]: osdmap e616: 12 total, 3 up, 11 in 2024-08-06T18:24:19.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/778147124' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:19.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:19.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:19.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:19.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:19.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:19.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:19 smithi107 ceph-mon[32381]: pgmap v1841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:19.737 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.738 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:19.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:19 smithi086 ceph-mon[29229]: pgmap v1841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:19.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:19 smithi117 ceph-mon[32835]: pgmap v1841: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:19.906 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:19.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:19.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:20.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:20.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:20.247 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:20.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:20.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:20.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:20.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:20.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:20.772 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:20.877 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:20.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:24:20.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:21.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:21.128 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:24:21.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:24:21.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:21.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:21.438 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:21.485 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:21.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:21.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:21.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:21.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:21.834 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:21.835 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:21.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:21 smithi086 ceph-mon[29229]: pgmap v1842: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:21.835 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:21 smithi086 ceph-mon[29229]: osdmap e617: 12 total, 3 up, 11 in 2024-08-06T18:24:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:21 smithi117 ceph-mon[32835]: pgmap v1842: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:21.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:21 smithi117 ceph-mon[32835]: osdmap e617: 12 total, 3 up, 11 in 2024-08-06T18:24:21.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:21.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:24:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:21 smithi107 ceph-mon[32381]: pgmap v1842: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:21.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:21 smithi107 ceph-mon[32381]: osdmap e617: 12 total, 3 up, 11 in 2024-08-06T18:24:22.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:22.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:22.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:22.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:22.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:22.621 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:22.656 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:22.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:22.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:22.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:23.097 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:23.098 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:23.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:23.186 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:23.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:24:23.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:23.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:23.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:23.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:23.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:23.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:23 smithi117 ceph-mon[32835]: pgmap v1844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:23.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:23.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:23 smithi086 ceph-mon[29229]: pgmap v1844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:24.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:24.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:23 smithi107 ceph-mon[32381]: pgmap v1844: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:24.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:24.323 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:24.323 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:24.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:24.465 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:24.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:24.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:24.737 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:24:24.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:24.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:24.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:24.867 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:24.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:24.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:25.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:25.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:25.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:25.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:25.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:25.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:25.822 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:25.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:25.823 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:25.839 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:25.840 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:25.840 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:25.840 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:25.840 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:25.857 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:25.858 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:25 smithi117 ceph-mon[32835]: pgmap v1845: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:25 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:25 smithi117 ceph-mon[32835]: osdmap e618: 12 total, 3 up, 11 in 2024-08-06T18:24:25.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:25 smithi107 ceph-mon[32381]: pgmap v1845: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:25 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:25 smithi107 ceph-mon[32381]: osdmap e618: 12 total, 3 up, 11 in 2024-08-06T18:24:26.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:26.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:25 smithi086 ceph-mon[29229]: pgmap v1845: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:26.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:25 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:26.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3793606138' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:26.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:25 smithi086 ceph-mon[29229]: osdmap e618: 12 total, 3 up, 11 in 2024-08-06T18:24:26.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:26.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:26.355 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:26.355 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:26.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:26.355 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:26.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:26.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:26.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:26.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:26.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:26.745 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:26.745 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:26.862 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:26.883 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:26.883 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:26.883 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:26.884 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: osdmap e619: 12 total, 3 up, 11 in 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1712306005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:26.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:26.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:27.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:24:27.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:24:27.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: osdmap e619: 12 total, 3 up, 11 in 2024-08-06T18:24:27.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1712306005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:27.018 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.019 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:24:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:24:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: osdmap e619: 12 total, 3 up, 11 in 2024-08-06T18:24:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1712306005' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:27.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:27.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:27.538 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:27.538 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:27.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:27.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.831 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:27 smithi086 ceph-mon[29229]: pgmap v1848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:27.832 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:27.832 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:27.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:27.960 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:27.961 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:27 smithi117 ceph-mon[32835]: pgmap v1848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:28.061 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:28.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:28.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:27 smithi107 ceph-mon[32381]: pgmap v1848: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:28.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:28.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:24:28.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:28.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:28.731 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:28.766 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:28.766 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:28.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:28.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:24:28.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:28.919 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:29.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:29.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:29.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:29.311 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:29.311 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:29.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:29.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:29.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:24:29.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:29.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:29.664 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:29.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:29.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:29.820 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:29 smithi086 ceph-mon[29229]: pgmap v1849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:29.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:29 smithi117 ceph-mon[32835]: pgmap v1849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:29.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:29 smithi107 ceph-mon[32381]: pgmap v1849: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:30.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:30.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:30.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:30.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:30.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:30.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:30.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:30.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:30.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:30.867 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:30.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:31.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:31.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:31.166 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:31.166 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:31.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:31.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:31.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:31.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:31.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:31.679 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:31.753 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:24:31.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:31.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:32.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:32.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:31 smithi117 ceph-mon[32835]: pgmap v1850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:32.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:32.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:31 smithi086 ceph-mon[29229]: pgmap v1850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:32.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:32.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:32.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:31 smithi107 ceph-mon[32381]: pgmap v1850: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:32.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:32.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:32.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:32.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:32.574 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:32.574 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:32.574 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:32.574 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:32.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:32.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:32 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:32.838 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:32 smithi086 ceph-mon[29229]: osdmap e620: 12 total, 3 up, 11 in 2024-08-06T18:24:32.889 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:33.002 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:33.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:32 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:32 smithi117 ceph-mon[32835]: osdmap e620: 12 total, 3 up, 11 in 2024-08-06T18:24:33.153 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:33.174 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:33.174 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:32 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2292166125' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:32 smithi107 ceph-mon[32381]: osdmap e620: 12 total, 3 up, 11 in 2024-08-06T18:24:33.282 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:33.417 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:33.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:33.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:33.691 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:33.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:24:33.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:33.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:33 smithi086 ceph-mon[29229]: pgmap v1852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:33.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:33 smithi107 ceph-mon[32381]: pgmap v1852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:33.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:33.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:34.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:33 smithi117 ceph-mon[32835]: pgmap v1852: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:34.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:34.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:34.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:34.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:34.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:34.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:24:34.613 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:34.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:34.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:34.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:34.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:34.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:35.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:35.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:35.192 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:35.192 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:35.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:35.301 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:35.301 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:35.302 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:35.307 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:35.307 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:35.307 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:35.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:35.308 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:35.327 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:35.327 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:35.327 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:35.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:35.435 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:35.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:35.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:35.593 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:35.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:35.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:35 smithi086 ceph-mon[29229]: pgmap v1853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/683162073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:35.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:35.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:35.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:35.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:35 smithi117 ceph-mon[32835]: pgmap v1853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:35.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/683162073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:35.995 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:35 smithi107 ceph-mon[32381]: pgmap v1853: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/683162073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:36.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.236 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:36.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:36.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:36.532 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:36.532 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:36.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:36.561 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:36.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:36.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:36.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.776 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:36 smithi117 ceph-mon[32835]: osdmap e621: 12 total, 3 up, 11 in 2024-08-06T18:24:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:36 smithi086 ceph-mon[29229]: osdmap e621: 12 total, 3 up, 11 in 2024-08-06T18:24:36.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:36.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:36.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:36 smithi107 ceph-mon[32381]: osdmap e621: 12 total, 3 up, 11 in 2024-08-06T18:24:37.032 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:37.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:37.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:37.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:37.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:37.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:37.702 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:37.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:37.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:38.102 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:37 smithi086 ceph-mon[29229]: pgmap v1854: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:38.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:38.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:38.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:38.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:37 smithi117 ceph-mon[32835]: pgmap v1854: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:37 smithi107 ceph-mon[32381]: pgmap v1854: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:38.232 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:38.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:38.486 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:38.486 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:38.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:38.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:24:38.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:38.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:38.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:38.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:38.791 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:38.791 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:38.804 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:38.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:39.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:39.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:39.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:39.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:39.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:39.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:39.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:39.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:39.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:24:39.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:39.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:39.586 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:39.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:39.702 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:39.702 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:39.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:39 smithi107 ceph-mon[32381]: pgmap v1856: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:39.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:39.988 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:39 smithi117 ceph-mon[32835]: pgmap v1856: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:39.998 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:40.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:39 smithi086 ceph-mon[29229]: pgmap v1856: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:40.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:40.263 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:40.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:40.500 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:40.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:40.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:40.783 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:24:40.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:41.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:41.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:41.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:41.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:41.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:41.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:24:41.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:41.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:41.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:41.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:41.463 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:41.463 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:41.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:41.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:41.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:41.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:41.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:41.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:41.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:42.112 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:42.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:42.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:41 smithi086 ceph-mon[29229]: pgmap v1857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:41 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:41 smithi086 ceph-mon[29229]: osdmap e622: 12 total, 3 up, 11 in 2024-08-06T18:24:42.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:41 smithi086 ceph-mon[29229]: osdmap e623: 12 total, 3 up, 11 in 2024-08-06T18:24:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:41 smithi117 ceph-mon[32835]: pgmap v1857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:41 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:41 smithi117 ceph-mon[32835]: osdmap e622: 12 total, 3 up, 11 in 2024-08-06T18:24:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:41 smithi117 ceph-mon[32835]: osdmap e623: 12 total, 3 up, 11 in 2024-08-06T18:24:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:41 smithi107 ceph-mon[32381]: pgmap v1857: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:41 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3057255038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:24:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:41 smithi107 ceph-mon[32381]: osdmap e622: 12 total, 3 up, 11 in 2024-08-06T18:24:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:41 smithi107 ceph-mon[32381]: osdmap e623: 12 total, 3 up, 11 in 2024-08-06T18:24:42.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:42.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:42.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:42.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:42.537 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:42.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:42.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:42.910 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:42.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:42 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:24:42.988 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:24:42.988 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:24:42.988 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:24:43.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:43.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:42 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:42 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:43.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:43.170 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:43.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:43.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:43.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:43.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:24:43.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:43.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:43 smithi086 ceph-mon[29229]: pgmap v1860: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:43 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:43.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:43.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:43 smithi117 ceph-mon[32835]: pgmap v1860: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:43.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:43.979 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:43 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:44.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:44.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:44.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:43 smithi107 ceph-mon[32381]: pgmap v1860: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:43 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:44.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:44.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:44.368 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:44.368 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:44.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:24:44.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:44.611 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:44.634 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:44.634 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:44.658 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:44.659 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:44.659 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:44.675 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:44.680 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:44.680 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:44.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:44.681 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:44.681 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:44.704 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:44.705 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:44.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:44.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:44.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:44.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:44.706 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:44.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:44.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:44.903 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:44 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:44.904 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3292476892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:44.904 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:44.904 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:44.957 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:45.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:45.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:45.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:45.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:44 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:45.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3292476892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:45.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:45.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:45.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:44 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3292476892' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:45.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:45.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:45.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:45.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:45.426 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:45.427 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:45.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:45.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:45.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:45.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:45 smithi117 ceph-mon[32835]: pgmap v1861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:45.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:45.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:45.848 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:45.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:45.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:45 smithi086 ceph-mon[29229]: pgmap v1861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:45.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:45.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:45.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:46.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:45 smithi107 ceph-mon[32381]: pgmap v1861: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:46.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:46.120 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:46.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:46.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:46.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:46.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:46.498 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:46.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:46.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:46.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:46.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:46.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:47.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:47.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:47.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:47.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:24:47.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:47.545 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:24:47.546 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:24:47.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:47.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:47.760 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:47.760 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:47.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:48.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:48.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:47 smithi117 ceph-mon[32835]: pgmap v1862: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:48.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:47 smithi086 ceph-mon[29229]: pgmap v1862: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:48.141 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:47 smithi107 ceph-mon[32381]: pgmap v1862: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:48.258 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:48.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:48.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:48.524 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:48.524 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:48.524 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:48.525 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:48.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:48.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:48.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:48.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:48.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:48.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:49.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:49.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:49.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:49.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:24:49.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:49.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:49.767 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:49.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:49.923 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:49 smithi117 ceph-mon[32835]: pgmap v1863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:50.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:50.055 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:49 smithi086 ceph-mon[29229]: pgmap v1863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:50.165 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:49 smithi107 ceph-mon[32381]: pgmap v1863: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:50.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:50.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:50.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:50.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:50.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:50.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:50.596 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:50.608 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:50.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:50.804 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:50.805 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:50.807 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:24:50.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:50.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:50.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:50.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:24:50.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:50.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:50.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:50.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:24:50.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:24:51.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:51.239 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:51.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:51.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:51.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:51.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:51.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: osdmap e624: 12 total, 3 up, 11 in 2024-08-06T18:24:51.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:51 smithi086 ceph-mon[29229]: osdmap e625: 12 total, 3 up, 11 in 2024-08-06T18:24:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:24:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: osdmap e624: 12 total, 3 up, 11 in 2024-08-06T18:24:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:51 smithi107 ceph-mon[32381]: osdmap e625: 12 total, 3 up, 11 in 2024-08-06T18:24:52.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1493302703' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: osdmap e624: 12 total, 3 up, 11 in 2024-08-06T18:24:52.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:51 smithi117 ceph-mon[32835]: osdmap e625: 12 total, 3 up, 11 in 2024-08-06T18:24:52.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:52.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:52.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:52.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:52.678 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:24:52.678 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:24:52.678 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:24:52.685 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:24:52.685 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:24:52.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:24:52.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:24:52.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:24:52.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:52.704 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:24:52.704 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:24:52.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:24:52.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:24:52.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:24:52.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:24:52.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:24:52.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:52.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3706836224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:52.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:52.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:52.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:53.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:53.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3706836224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3706836224' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:24:53.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:53.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:53.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:53.596 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:24:53.596 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:24:53.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:53.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:53.809 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:24:53.809 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:509,from:2,reservation_nonce:20)", 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 456.64920190200002, 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.0011134249999999999, 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.810 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160994+0000", 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.811 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.160996+0000", 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5099999999999999e-06 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161004+0000", 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2330000000000006e-06 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161020+0000", 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5559000000000001e-05 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.812 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161028+0000", 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2819999999999996e-06 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161088+0000", 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9743e-05 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "op must wait for map", 2024-08-06T18:24:53.813 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.161092+0000", 2024-08-06T18:24:53.814 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7160000000000002e-06 2024-08-06T18:24:53.814 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162061+0000", 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00096807299999999996 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.815 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.816 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.816 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162076+0000", 2024-08-06T18:24:53.816 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5928999999999999e-05 2024-08-06T18:24:53.816 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162108+0000", 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1380000000000001e-05 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.817 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:509/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 456.64761663199999, 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112194, 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.818 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162580+0000", 2024-08-06T18:24:53.819 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8400000000000004e-07 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162592+0000", 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1977000000000001e-05 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162605+0000", 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2602000000000001e-05 2024-08-06T18:24:53.820 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162614+0000", 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.918e-06 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162663+0000", 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9168000000000001e-05 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.821 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162670+0000", 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.396e-06 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.162692+0000", 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1449000000000001e-05 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:509,from:2,reservation_nonce:0)", 2024-08-06T18:24:53.822 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 456.646972764, 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112296, 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.823 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163224+0000", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9499999999999998e-07 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163226+0000", 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.1940000000000001e-06 2024-08-06T18:24:53.824 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163231+0000", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1819999999999997e-06 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163235+0000", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8290000000000001e-06 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163305+0000", 2024-08-06T18:24:53.825 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0358000000000006e-05 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163322+0000", 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6384000000000001e-05 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:17:17.163336+0000", 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4054e-05 2024-08-06T18:24:53.826 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:553,from:2,reservation_nonce:22)", 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 271.42258858100001, 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012249200000000001, 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.827 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387609+0000", 2024-08-06T18:24:53.828 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.48e-06 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387617+0000", 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9510000000000006e-06 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387631+0000", 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4141000000000001e-05 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.829 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387640+0000", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3799999999999994e-06 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387689+0000", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9407000000000003e-05 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387705+0000", 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5583e-05 2024-08-06T18:24:53.830 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387730+0000", 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5550000000000001e-05 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:553/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 271.42202902899999, 2024-08-06T18:24:53.831 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012141, 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.832 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388168+0000", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9500000000000002e-07 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388172+0000", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2220000000000004e-06 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388181+0000", 2024-08-06T18:24:53.833 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0610000000000002e-06 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388188+0000", 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.153e-06 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388247+0000", 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8544000000000001e-05 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.834 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388258+0000", 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0879999999999999e-05 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388289+0000", 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0954999999999998e-05 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:553,from:2,reservation_nonce:0)", 2024-08-06T18:24:53.835 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 271.42135032700003, 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110738, 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.836 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4999999999999999e-07 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388848+0000", 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7349999999999999e-06 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.837 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388854+0000", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5450000000000003e-06 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388859+0000", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0520000000000001e-06 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388937+0000", 2024-08-06T18:24:53.838 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8552000000000003e-05 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388949+0000", 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1409e-05 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388957+0000", 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1950000000000003e-06 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.839 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:553,from:6,reservation_nonce:24)", 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 268.40595586000001, 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014068199999999999, 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:24:53.840 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404242+0000", 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5540000000000001e-06 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.841 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404251+0000", 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2010000000000005e-06 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404270+0000", 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9028000000000001e-05 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404279+0000", 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5220000000000007e-06 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.842 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.843 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.843 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404340+0000", 2024-08-06T18:24:53.843 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1649000000000006e-05 2024-08-06T18:24:53.843 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.843 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404355+0000", 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4484999999999999e-05 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404381+0000", 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6242999999999999e-05 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.844 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:553/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 268.40521305200002, 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101147, 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:24:53.845 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404984+0000", 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.58e-07 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.846 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404995+0000", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.095e-05 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405004+0000", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3200000000000006e-06 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405009+0000", 2024-08-06T18:24:53.847 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0930000000000002e-06 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405053+0000", 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3495000000000003e-05 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405060+0000", 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4619999999999999e-06 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.848 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405084+0000", 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4369e-05 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:575,from:6,reservation_nonce:25)", 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 186.23256504, 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020588599999999999, 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.849 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.850 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577634+0000", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2400000000000002e-06 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577646+0000", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2357e-05 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577680+0000", 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4119000000000003e-05 2024-08-06T18:24:53.851 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577696+0000", 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5716e-05 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577796+0000", 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010001900000000001 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.852 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577811+0000", 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5299e-05 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577837+0000", 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6135999999999998e-05 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:575/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:24:53.853 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 186.23141654, 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028445800000000002, 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.854 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578786+0000", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5400000000000001e-06 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578798+0000", 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1236000000000001e-05 2024-08-06T18:24:53.855 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578827+0000", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9070000000000001e-05 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578842+0000", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4916000000000001e-05 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578978+0000", 2024-08-06T18:24:53.856 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136534 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578998+0000", 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9412999999999999e-05 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.579064+0000", 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6748999999999994e-05 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.857 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:575,from:6,reservation_nonce:0)", 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 186.230099373, 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016099300000000001, 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:24:53.858 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.859 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.859 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.859 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:53.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580098+0000", 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3300000000000002e-07 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580101+0000", 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6940000000000001e-06 2024-08-06T18:24:53.860 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580130+0000", 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8833e-05 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580136+0000", 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7520000000000002e-06 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.861 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580238+0000", 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102543 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580249+0000", 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0823e-05 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580258+0000", 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8149999999999995e-06 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.862 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:599,from:6,reservation_nonce:26)", 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 86.028172436000006, 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018547099999999999, 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.863 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782025+0000", 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5340000000000001e-06 2024-08-06T18:24:53.864 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782035+0000", 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999993e-06 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782055+0000", 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0650999999999999e-05 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.865 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782064+0000", 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9209999999999998e-06 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782156+0000", 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1822000000000003e-05 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782172+0000", 2024-08-06T18:24:53.866 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5611999999999999e-05 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782209+0000", 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7830999999999997e-05 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:599/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 86.027531084000003, 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111396, 2024-08-06T18:24:53.867 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.868 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782666+0000", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6899999999999998e-07 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782670+0000", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.048e-06 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782697+0000", 2024-08-06T18:24:53.869 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6948999999999999e-05 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782702+0000", 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8199999999999996e-06 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782739+0000", 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7166999999999998e-05 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.870 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782745+0000", 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1430000000000003e-06 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782777+0000", 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.18e-05 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.871 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:599,from:6,reservation_nonce:0)", 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 86.026838049999995, 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136124, 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:24:53.872 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783359+0000", 2024-08-06T18:24:53.873 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1699999999999999e-07 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783361+0000", 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5359999999999999e-06 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783367+0000", 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.285e-06 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.874 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783376+0000", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4780000000000008e-06 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783463+0000", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6978999999999995e-05 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783480+0000", 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6634e-05 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:24:53.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783494+0000", 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4895000000000001e-05 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:24:53.876 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:24:53.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:53.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:24:54.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:54.036 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:54.037 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:54.037 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:53 smithi107 ceph-mon[32381]: pgmap v1867: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:54.037 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.037 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.037 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:53 smithi086 ceph-mon[29229]: pgmap v1867: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:54.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:53 smithi117 ceph-mon[32835]: pgmap v1867: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:24:54.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:54.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:54.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:54.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:54.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:54.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:54.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:54.730 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:54.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:24:54.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:54.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:54.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:54.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:55.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:55.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:55.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:55.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:55.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:24:55.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:55.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:55.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:55.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:55.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:55.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:55.878 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:55 smithi086 ceph-mon[29229]: pgmap v1868: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:55.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:56.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:55 smithi117 ceph-mon[32835]: pgmap v1868: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:56.159 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:56.194 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:56.194 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:55 smithi107 ceph-mon[32381]: pgmap v1868: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:56.296 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:56.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:56.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:56.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:56.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:24:56.837 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:56.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:57.069 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:57.070 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:57.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:24:57.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:57.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:57.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:57.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:24:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:57.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:57.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:57.819 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:24:57.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:57.943 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:24:57.943 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:24:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:24:57.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:57 smithi086 ceph-mon[29229]: pgmap v1869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:57.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:58.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:24:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:57 smithi117 ceph-mon[32835]: pgmap v1869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:58.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:24:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:57 smithi107 ceph-mon[32381]: pgmap v1869: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:24:58.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:58.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:58.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:24:58.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:58.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:24:58.888 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:58 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:58.889 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:58.889 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:58 smithi107 ceph-mon[32381]: osdmap e626: 12 total, 3 up, 11 in 2024-08-06T18:24:58.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:24:59.012 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:58 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:59.012 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:59.012 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:58 smithi117 ceph-mon[32835]: osdmap e626: 12 total, 3 up, 11 in 2024-08-06T18:24:59.029 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:24:59.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:24:59.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:24:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:24:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:58 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:24:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3358701067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:24:59.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:58 smithi086 ceph-mon[29229]: osdmap e626: 12 total, 3 up, 11 in 2024-08-06T18:24:59.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:24:59.258 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:59.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:24:59.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:59.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:59.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:24:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:24:59.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:24:59.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:24:59.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:24:59.824 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:24:59.824 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:24:59.863 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:24:59 smithi086 ceph-mon[29229]: pgmap v1871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:24:59.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:00.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:00.079 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:24:59 smithi107 ceph-mon[32381]: pgmap v1871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:00.079 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:00.079 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:24:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:24:59 smithi117 ceph-mon[32835]: pgmap v1871: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:00.188 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:00.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:00.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:00.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:25:00.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:00.737 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:00.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:00.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:00.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:00.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:01.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:01.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:01.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:01.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:01.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:01.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:01.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:01.836 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:01.837 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:01.837 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:01.854 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:01.855 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:01.872 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:01.872 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:01.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:01.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:01.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:01.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:01.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:01.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:02.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:01 smithi086 ceph-mon[29229]: pgmap v1872: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:02.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:01 smithi086 ceph-mon[29229]: osdmap e627: 12 total, 3 up, 11 in 2024-08-06T18:25:02.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2910688592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:02.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:02.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:01 smithi107 ceph-mon[32381]: pgmap v1872: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:02.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:01 smithi107 ceph-mon[32381]: osdmap e627: 12 total, 3 up, 11 in 2024-08-06T18:25:02.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2910688592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:02.095 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:02.095 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:01 smithi117 ceph-mon[32835]: pgmap v1872: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:01 smithi117 ceph-mon[32835]: osdmap e627: 12 total, 3 up, 11 in 2024-08-06T18:25:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2910688592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:02.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:02.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:02.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:02.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:02.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:02.853 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:02.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:02.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:02.903 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:03.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:03.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:03.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:03.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:03.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:03.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:03.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:03.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:03.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:25:03.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:03.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:03.722 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:03.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:03.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:04.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:04.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:04.103 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:03 smithi086 ceph-mon[29229]: pgmap v1874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:04.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:04.114 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:03 smithi107 ceph-mon[32381]: pgmap v1874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:04.114 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:04.115 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:03 smithi117 ceph-mon[32835]: pgmap v1874: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:04.129 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:04.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:04.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:04.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:04.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:04.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:04.877 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:04.891 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:05.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:05.161 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:05.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:05.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:05.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:05.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:05.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:05.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:06.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:05 smithi086 ceph-mon[29229]: pgmap v1875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:06.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:05 smithi117 ceph-mon[32835]: pgmap v1875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:06.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:25:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:05 smithi107 ceph-mon[32381]: pgmap v1875: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:06.234 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:06.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:06.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:06.629 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:06.728 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:06.728 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:06.728 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:06.728 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:06.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:06.841 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:25:06.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:06.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:07.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:07.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:07.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:07.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:07.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:07.427 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:07.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:07.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:07.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:07.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:07.739 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:07.833 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:07.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:07 smithi117 ceph-mon[32835]: pgmap v1876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:07.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:07 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:07.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:07.862 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:07 smithi117 ceph-mon[32835]: osdmap e628: 12 total, 3 up, 11 in 2024-08-06T18:25:07.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:07 smithi086 ceph-mon[29229]: pgmap v1876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:07.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:07 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:07.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:07.963 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:07 smithi086 ceph-mon[29229]: osdmap e628: 12 total, 3 up, 11 in 2024-08-06T18:25:07.971 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:08.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:08.131 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:07 smithi107 ceph-mon[32381]: pgmap v1876: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:08.131 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:07 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:08.131 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4234984146' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:08.131 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:07 smithi107 ceph-mon[32381]: osdmap e628: 12 total, 3 up, 11 in 2024-08-06T18:25:08.206 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:08.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:08.374 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:08.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:08.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:08.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:08.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:25:08.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:09.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:09.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:09.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:09.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:09.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:09.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:09.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:09.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:25:09.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:09.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:09.908 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:09 smithi107 ceph-mon[32381]: pgmap v1878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:10.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:10.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:09 smithi086 ceph-mon[29229]: pgmap v1878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:10.038 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:10.038 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:10.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:10.038 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:09 smithi117 ceph-mon[32835]: pgmap v1878: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:10.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:10.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:10.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:10.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:10.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:10.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:10.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:10.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:10.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:11.020 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:11.050 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:11.050 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:11.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:11.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:11.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:11.085 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:11.085 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:11.086 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:11.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:11.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:11.249 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:11.249 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:11.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:11.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:11.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:11.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:11.843 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:11.843 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:11.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:11.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:11.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:11 smithi107 ceph-mon[32381]: pgmap v1879: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2152575196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:11.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:11 smithi107 ceph-mon[32381]: osdmap e629: 12 total, 3 up, 11 in 2024-08-06T18:25:12.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:12.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:11 smithi086 ceph-mon[29229]: pgmap v1879: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2152575196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:12.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:11 smithi086 ceph-mon[29229]: osdmap e629: 12 total, 3 up, 11 in 2024-08-06T18:25:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:11 smithi117 ceph-mon[32835]: pgmap v1879: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2152575196' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:11 smithi117 ceph-mon[32835]: osdmap e629: 12 total, 3 up, 11 in 2024-08-06T18:25:12.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:12.226 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:12.226 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:12.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:12.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:12.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:12.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:12.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:12.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:13.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:13.113 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:13.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:13.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:13.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:13.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:13.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:13.367 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:13.503 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:13.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:13.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:13.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:14.035 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:25:14.035 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:25:14.035 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:25:14.035 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:25:14.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:14.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:13 smithi117 ceph-mon[32835]: pgmap v1881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:14.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:13 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:25:14.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:13 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:25:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:13 smithi086 ceph-mon[29229]: pgmap v1881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:13 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:25:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:13 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:25:14.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:14.202 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:13 smithi107 ceph-mon[32381]: pgmap v1881: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:14.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:13 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:25:14.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:13 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:25:14.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:25:14.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:14.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:14.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:14.600 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:14.600 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:14.709 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:14.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:25:14.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:14.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:15.102 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:15.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:14 smithi086 ceph-mon[29229]: osdmap e630: 12 total, 3 up, 11 in 2024-08-06T18:25:15.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:15.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:14 smithi117 ceph-mon[32835]: osdmap e630: 12 total, 3 up, 11 in 2024-08-06T18:25:15.147 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:15.186 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:14 smithi107 ceph-mon[32381]: osdmap e630: 12 total, 3 up, 11 in 2024-08-06T18:25:15.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:15.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:15.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:15.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:15.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:15.860 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:25:15.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:15 smithi086 ceph-mon[29229]: pgmap v1883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:15.868 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:15.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:16.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:15 smithi117 ceph-mon[32835]: pgmap v1883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:16.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:15 smithi107 ceph-mon[32381]: pgmap v1883: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:16.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:16.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:16.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:16.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:16.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:16.990 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:17.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:17.148 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:16 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:17.148 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:17.148 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:16 smithi086 ceph-mon[29229]: osdmap e631: 12 total, 3 up, 11 in 2024-08-06T18:25:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:16 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:16 smithi107 ceph-mon[32381]: osdmap e631: 12 total, 3 up, 11 in 2024-08-06T18:25:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:17.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:17.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:16 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1834842264' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:17.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:16 smithi117 ceph-mon[32835]: osdmap e631: 12 total, 3 up, 11 in 2024-08-06T18:25:17.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:17.527 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:17.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:17.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:17.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:17 smithi107 ceph-mon[32381]: pgmap v1885: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:18.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:17 smithi086 ceph-mon[29229]: pgmap v1885: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:18.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:18.224 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:17 smithi117 ceph-mon[32835]: pgmap v1885: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:18.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:18.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:18.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:18.619 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:18.620 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:18.620 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:18.628 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:18.628 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:18.636 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:18.637 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:18.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:18.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:18.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:18.661 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:18.661 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:18.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:18.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:18.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:18.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:18.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:18.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:18.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:18.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1199215599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:18.879 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:19.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:19.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1199215599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:19.142 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:19.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:19.168 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1199215599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:19.168 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:19.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:19.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:19.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:19.605 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:19.606 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:19.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:19.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:25:19.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:19.901 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:25:19.901 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:25:19.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:19 smithi107 ceph-mon[32381]: pgmap v1886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:19.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:19.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:20.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:19 smithi086 ceph-mon[29229]: pgmap v1886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:20.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:20.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:20.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:19 smithi117 ceph-mon[32835]: pgmap v1886: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:20.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:20.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:20.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:20.592 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:20.592 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:20.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:20.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:25:20.629 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:20.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:20.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:21.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:21.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:21.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:21.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:21.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:21.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:21.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:21.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:21.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:25:21.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:21.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:21.845 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:21.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:22.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:22.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:21 smithi086 ceph-mon[29229]: pgmap v1887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:22.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:22.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:21 smithi107 ceph-mon[32381]: pgmap v1887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:22.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:22.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:21 smithi117 ceph-mon[32835]: pgmap v1887: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:22.499 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:22.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:22.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:22.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:23.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:23.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:23.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:23.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:23.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:23.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:23.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:23.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:23.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:23.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:23.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:23.708 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:23.836 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:23.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:23.847 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:25:23.847 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:25:23.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:23.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:23.886 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:23.886 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:23.891 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:25:23.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:24.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:24.142 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:23 smithi107 ceph-mon[32381]: pgmap v1888: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:24.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:24.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:23 smithi086 ceph-mon[29229]: pgmap v1888: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:24.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:24.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:23 smithi117 ceph-mon[32835]: pgmap v1888: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:24.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:24.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:24.610 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:25:24.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:24.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:24.872 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:24.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:24.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:24 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:24.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:24.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:24 smithi117 ceph-mon[32835]: osdmap e632: 12 total, 3 up, 11 in 2024-08-06T18:25:25.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:25.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:24 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:25.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:25.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:24 smithi107 ceph-mon[32381]: osdmap e632: 12 total, 3 up, 11 in 2024-08-06T18:25:25.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:25.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:24 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:25.194 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1126558281' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:25.194 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:24 smithi086 ceph-mon[29229]: osdmap e632: 12 total, 3 up, 11 in 2024-08-06T18:25:25.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:25.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:25.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:25.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:25.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:25.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:26.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:26.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:25 smithi107 ceph-mon[32381]: pgmap v1890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:26.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:26.280 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:25 smithi086 ceph-mon[29229]: pgmap v1890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:25 smithi117 ceph-mon[32835]: pgmap v1890: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:26.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:26.516 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:26.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:26.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:26.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:26.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:26.993 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:26.994 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:27.009 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:27.010 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:27.026 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:27.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:27.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:27.102 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:27.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:25:27.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:27.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:27.359 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:27.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:27.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:27.727 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:27.727 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:27.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:27.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:27.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:27 smithi107 ceph-mon[32381]: pgmap v1891: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:27.882 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:27 smithi107 ceph-mon[32381]: osdmap e633: 12 total, 3 up, 11 in 2024-08-06T18:25:27.882 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/273527083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:27.882 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:27.882 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:27.892 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:27.893 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:27.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:28.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:27 smithi086 ceph-mon[29229]: pgmap v1891: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:28.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:27 smithi086 ceph-mon[29229]: osdmap e633: 12 total, 3 up, 11 in 2024-08-06T18:25:28.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/273527083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:28.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:28.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:28.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:27 smithi117 ceph-mon[32835]: pgmap v1891: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:27 smithi117 ceph-mon[32835]: osdmap e633: 12 total, 3 up, 11 in 2024-08-06T18:25:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/273527083' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:28.251 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:28.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:28.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:28.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:28.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:25:28.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:28.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:28.745 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:28.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:28.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:28.912 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:28.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:29.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:29.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:29.204 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:29.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:29.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:29.565 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:29.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:25:29.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:29.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:29.812 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:29.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:29.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:29 smithi107 ceph-mon[32381]: pgmap v1893: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:29.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:25:29.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:29.998 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:29 smithi117 ceph-mon[32835]: pgmap v1893: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:30.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:29 smithi086 ceph-mon[29229]: pgmap v1893: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:30.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:30.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:30.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:30.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:30.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:30.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:30.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:30.606 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:30.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:30.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:30 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:25:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:30 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:25:30.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:30 smithi117 ceph-mon[32835]: osdmap e634: 12 total, 3 up, 11 in 2024-08-06T18:25:30.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:30 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:25:30.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:30 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:25:30.896 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:30 smithi107 ceph-mon[32381]: osdmap e634: 12 total, 3 up, 11 in 2024-08-06T18:25:30.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:31.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:30 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:25:31.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:30 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:25:31.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:30 smithi086 ceph-mon[29229]: osdmap e634: 12 total, 3 up, 11 in 2024-08-06T18:25:31.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:31.149 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:31.149 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:31.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:31.301 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:31.323 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:31.323 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:31.341 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:31.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:31.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:31.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:31.726 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:25:31.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:31 smithi107 ceph-mon[32381]: pgmap v1895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:31 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:31 smithi107 ceph-mon[32381]: osdmap e635: 12 total, 3 up, 11 in 2024-08-06T18:25:31.969 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:31.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:31.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:31.991 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:31 smithi086 ceph-mon[29229]: pgmap v1895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:31.991 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:31.991 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:31 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:31.991 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:31.991 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:31 smithi086 ceph-mon[29229]: osdmap e635: 12 total, 3 up, 11 in 2024-08-06T18:25:32.028 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:32.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:25:32.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:31 smithi117 ceph-mon[32835]: pgmap v1895: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 329 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:31 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2865246638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:32.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:31 smithi117 ceph-mon[32835]: osdmap e635: 12 total, 3 up, 11 in 2024-08-06T18:25:32.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:32.282 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:32.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:32.292 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:32.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:32.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:32.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:32.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:32.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/59633210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:32.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:32.925 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/59633210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:33.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:33.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:33.067 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:33.068 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:33.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:33.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:33.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/59633210' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:33.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:33.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:33.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:33.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:33.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:33.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:33.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:33.761 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:33.762 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:33.781 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:33.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:33.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:33 smithi107 ceph-mon[32381]: pgmap v1897: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:33.983 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:33 smithi117 ceph-mon[32835]: pgmap v1897: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:33.983 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:33.983 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:33.983 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:34.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:33 smithi086 ceph-mon[29229]: pgmap v1897: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:34.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:34.291 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:34.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:25:34.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:34.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:34.543 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:34.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:34.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:34.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:34.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:34.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:34.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:25:35.013 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:35.013 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:35.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:35.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:35.245 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:35.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:35.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:35.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:35.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:35.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:25:35.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:35.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:35.975 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:35.975 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:35.975 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:35 smithi107 ceph-mon[32381]: pgmap v1898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:36.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:35 smithi086 ceph-mon[29229]: pgmap v1898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:36.083 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:36.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:35 smithi117 ceph-mon[32835]: pgmap v1898: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:36.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:36.367 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:36.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:36.550 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:36.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:36.657 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:36.657 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:36.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:25:36.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:36.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:36.955 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:37.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:37.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:37.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:37.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:37.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:37.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:37.762 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:25:37.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:37.970 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:38.017 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:37 smithi107 ceph-mon[32381]: pgmap v1899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:38.017 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:38.105 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:37 smithi086 ceph-mon[29229]: pgmap v1899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:38.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:38.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:37 smithi117 ceph-mon[32835]: pgmap v1899: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:38.236 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:38.304 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:38.304 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:38.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:38.497 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:38.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:38.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:38.757 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:38.757 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:38.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:38.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:39.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:39.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:38 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:39.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:39.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:38 smithi107 ceph-mon[32381]: osdmap e636: 12 total, 3 up, 11 in 2024-08-06T18:25:39.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:38 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:39.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:39.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:38 smithi117 ceph-mon[32835]: osdmap e636: 12 total, 3 up, 11 in 2024-08-06T18:25:39.059 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:39.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:25:39.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:39.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:39.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:39.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:38 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1564382203' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:39.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:38 smithi086 ceph-mon[29229]: osdmap e636: 12 total, 3 up, 11 in 2024-08-06T18:25:39.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:39.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:39.306 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:39.373 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:39.373 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:39.373 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:39.390 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:39.390 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:39.391 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:39.391 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:39.391 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:39.407 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:39.408 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:39.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:39.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:39.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:39.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:39.409 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:39.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:39.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:39.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:39.685 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:39.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:39.824 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:39.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:39 smithi107 ceph-mon[32381]: pgmap v1901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:39.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4051045689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:39.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:39 smithi086 ceph-mon[29229]: pgmap v1901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:39.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4051045689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:39.978 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:39.978 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:40.079 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:40.079 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:40.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:40.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:39 smithi117 ceph-mon[32835]: pgmap v1901: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:40.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4051045689' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:40.222 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:40.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:40.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:40.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:40.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:40.562 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:40.562 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:40.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:40.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:40.792 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:25:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:25:40.792 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:25:40.792 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:25:40.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:40.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:40.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:40.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:41.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:41.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:41.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:41.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:25:41.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:41.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:41.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:25:41.441 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:41.441 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:41.449 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:41.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:41.685 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:41.688 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:41.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:41 smithi086 ceph-mon[29229]: pgmap v1902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:41.830 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:41 smithi086 ceph-mon[29229]: osdmap e637: 12 total, 3 up, 11 in 2024-08-06T18:25:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:41 smithi117 ceph-mon[32835]: pgmap v1902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:41.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:41 smithi117 ceph-mon[32835]: osdmap e637: 12 total, 3 up, 11 in 2024-08-06T18:25:41.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:42.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:41 smithi107 ceph-mon[32381]: pgmap v1902: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:42.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:41 smithi107 ceph-mon[32381]: osdmap e637: 12 total, 3 up, 11 in 2024-08-06T18:25:42.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:25:42.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:42.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:42.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:42.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:42.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:42.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:42.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:42.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:42.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:43.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:43.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:43.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:43.214 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:25:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:25:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:25:43.214 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:25:43.214 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:25:43.282 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:43.285 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:25:43.288+0000 7f87737fe640 0 --2- 172.21.15.86:0/4022485710 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f877410cd50 0x7f877419de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:25:43.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:43.549 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:43.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:43.774 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:25:43.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:25:43.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:43.980 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:44.061 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:44.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:43 smithi086 ceph-mon[29229]: pgmap v1904: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:43 smithi117 ceph-mon[32835]: pgmap v1904: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:44.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:43 smithi107 ceph-mon[32381]: pgmap v1904: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:44.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:25:44.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:44.184 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:44.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:44.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:44.526 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:44.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:44.798 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:44.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:44.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:44 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:44.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:44.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:44 smithi117 ceph-mon[32835]: osdmap e638: 12 total, 3 up, 11 in 2024-08-06T18:25:45.039 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:44 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:45.039 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:45.040 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:44 smithi107 ceph-mon[32381]: osdmap e638: 12 total, 3 up, 11 in 2024-08-06T18:25:45.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:44 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4022485710' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:25:45.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:44 smithi086 ceph-mon[29229]: osdmap e638: 12 total, 3 up, 11 in 2024-08-06T18:25:45.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:45.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:45.463 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:45.464 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:45.465 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:45.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:45.483 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:45.483 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:45.483 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:45.483 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:45.483 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:45.499 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:45.500 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:45.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:45.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:45.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:45.501 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:45.501 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:45.569 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:25:45.570 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:25:45.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:45.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:45.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:45.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:45.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:45.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:45 smithi117 ceph-mon[32835]: pgmap v1906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:45.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2772578442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:46.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:45 smithi086 ceph-mon[29229]: pgmap v1906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2772578442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:46.129 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:45 smithi107 ceph-mon[32381]: pgmap v1906: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:46.129 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2772578442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:46.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:46.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:25:46.270 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:46.271 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:46.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:46.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:46.463 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:46.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:46.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:46.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:46.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:46.874 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:46.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:46.875 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:46 smithi107 ceph-mon[32381]: osdmap e639: 12 total, 3 up, 11 in 2024-08-06T18:25:46.944 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:47.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:47.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.100 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:46 smithi086 ceph-mon[29229]: osdmap e639: 12 total, 3 up, 11 in 2024-08-06T18:25:47.100 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:47.100 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:46 smithi117 ceph-mon[32835]: osdmap e639: 12 total, 3 up, 11 in 2024-08-06T18:25:47.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:25:47.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:47.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:47.445 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:47.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:47.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:47.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:47.767 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:47.768 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:47.852 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:47 smithi086 ceph-mon[29229]: pgmap v1907: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:47.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:47.903 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:47 smithi117 ceph-mon[32835]: pgmap v1907: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:47.939 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:48.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:48.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:48.165 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:47 smithi107 ceph-mon[32381]: pgmap v1907: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:48.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:25:48.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:48.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:48.527 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:48.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:25:48.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:48.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:48.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:48.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:48.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:48.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:48.775 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:48.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:48.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:49.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:49.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:49.224 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:49.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:49.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:49.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:49.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:49.804 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:25:49.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:49.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:49 smithi086 ceph-mon[29229]: pgmap v1909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:49.896 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:49.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:50.104 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:49 smithi107 ceph-mon[32381]: pgmap v1909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:50.105 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:50.105 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:50.105 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:49 smithi117 ceph-mon[32835]: pgmap v1909: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:25:50.128 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:50.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:50.322 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:50.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:50.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:50.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:50.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:50.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: osdmap e640: 12 total, 3 up, 11 in 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:50 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:50.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:50.981 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:50.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:25:51.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: osdmap e640: 12 total, 3 up, 11 in 2024-08-06T18:25:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:25:51.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:50 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:25:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2277133763' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:25:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: osdmap e640: 12 total, 3 up, 11 in 2024-08-06T18:25:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:25:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:50 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.241 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:25:51.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:51.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:51.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:51.586 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:51.586 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:51.586 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:51.607 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:51.607 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:51.607 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:51.608 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:51.608 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:51.608 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:51.608 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:51.625 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:51.626 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:51.626 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:51.626 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:51.626 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:51.626 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:51.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:51.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:51.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: pgmap v1911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1754293977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:51 smithi086 ceph-mon[29229]: osdmap e641: 12 total, 3 up, 11 in 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: pgmap v1911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1754293977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:51.912 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:51 smithi107 ceph-mon[32381]: osdmap e641: 12 total, 3 up, 11 in 2024-08-06T18:25:51.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:52.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: pgmap v1911: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1754293977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:25:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:25:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:25:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:51 smithi117 ceph-mon[32835]: osdmap e641: 12 total, 3 up, 11 in 2024-08-06T18:25:52.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:52.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:52.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:52.320 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:52.320 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:52.347 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:52.347 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:52.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:52.504 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:52.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:25:52.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:52.657 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:52.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:52.885 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:52.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:53.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:53.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:53.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:53.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:53.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:53.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:53.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:25:53.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:53.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:53.333 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:53.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:53.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:53.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:53.481 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:25:53.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:53.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:53.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:53.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:53.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:53.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:53 smithi117 ceph-mon[32835]: pgmap v1913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:53 smithi107 ceph-mon[32381]: pgmap v1913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:53.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:54.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:53 smithi086 ceph-mon[29229]: pgmap v1913: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:54.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:54.013 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:54.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:54.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:54.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:54.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:54.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:25:54.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:54.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:54.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:54.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:54.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:54.769 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:54.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:54.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:55.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:55.052 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:55.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:55.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:55.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:55.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:55.689 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:55.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:25:55.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:55.740 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:55.740 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:55.833 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:25:55.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:25:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:55 smithi086 ceph-mon[29229]: pgmap v1914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:55.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:55 smithi117 ceph-mon[32835]: pgmap v1914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:55.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:55.937 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:25:55.984 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:56.116 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:55 smithi107 ceph-mon[32381]: pgmap v1914: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:56.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:25:56.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:56.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:56.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:25:56.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:25:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:25:56.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:25:56.332 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:56.333 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:56.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:56.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:56.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:56.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:56.909 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:57.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:56 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:57.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:57.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:56 smithi117 ceph-mon[32835]: osdmap e642: 12 total, 3 up, 11 in 2024-08-06T18:25:57.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:56 smithi117 ceph-mon[32835]: osdmap e643: 12 total, 3 up, 11 in 2024-08-06T18:25:57.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:25:57.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:57.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:56 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:56 smithi086 ceph-mon[29229]: osdmap e642: 12 total, 3 up, 11 in 2024-08-06T18:25:57.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:56 smithi086 ceph-mon[29229]: osdmap e643: 12 total, 3 up, 11 in 2024-08-06T18:25:57.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:57.177 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:25:57.178 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:25:57.178 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:25:57.196 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:25:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:25:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:25:57.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:25:57.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:25:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:56 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:25:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1599022554' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:25:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:56 smithi107 ceph-mon[32381]: osdmap e642: 12 total, 3 up, 11 in 2024-08-06T18:25:57.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:56 smithi107 ceph-mon[32381]: osdmap e643: 12 total, 3 up, 11 in 2024-08-06T18:25:57.215 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:25:57.215 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:25:57.216 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:25:57.216 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:25:57.216 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:25:57.216 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:25:57.217 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:25:57.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:57.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:57.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:25:57.515 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:57.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:57.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:57.839 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:57.863 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:57 smithi107 ceph-mon[32381]: pgmap v1916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:57.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3653092707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:57.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:57.864 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:57.959 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:25:57.960 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:25:57.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:58.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:57 smithi117 ceph-mon[32835]: pgmap v1916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:58.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3653092707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:58.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:58.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:58.096 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:25:58.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:25:58.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:57 smithi086 ceph-mon[29229]: pgmap v1916: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3653092707' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:25:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:58.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:25:58.353 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:25:58.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:58.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:58.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:58.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:25:58.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:25:58.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:58.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:58.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:58.820 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:58.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:25:59.048 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:25:59.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:59.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:59.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:25:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:25:59.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:59.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:59.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:59.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:25:59.345 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:25:59.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:25:59.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:25:59.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:25:59.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:25:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:25:59.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:25:59.837 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:25:59.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:25:59.960 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:25:59.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:25:59 smithi107 ceph-mon[32381]: pgmap v1918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:25:59.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:00.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:25:59 smithi086 ceph-mon[29229]: pgmap v1918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:25:59 smithi117 ceph-mon[32835]: pgmap v1918: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:00.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:00.265 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:00.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:00.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:00.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:00.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:00.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:26:00.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:00.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:00.661 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:00.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:00.808 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:00.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:01.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:01.183 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:01.314 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:01.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:01.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:01.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:01.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:01.741 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:01.741 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:01.850 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:26:01.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:01.890 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:01 smithi117 ceph-mon[32835]: pgmap v1919: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:01.890 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:01.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:02.009 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:01 smithi086 ceph-mon[29229]: pgmap v1919: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:02.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:02.075 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:02.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:02.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:01 smithi107 ceph-mon[32381]: pgmap v1919: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:02.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:02.261 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:02.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:02.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:02.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:02.670 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:02.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:26:02.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:02.684 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:02.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:02.920 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:02.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:02.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:02 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:02.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:02.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:02 smithi086 ceph-mon[29229]: osdmap e644: 12 total, 3 up, 11 in 2024-08-06T18:26:02.947 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:02 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:02.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:02.948 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:02 smithi117 ceph-mon[32835]: osdmap e644: 12 total, 3 up, 11 in 2024-08-06T18:26:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:02 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3140945090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:02.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:02 smithi107 ceph-mon[32381]: osdmap e644: 12 total, 3 up, 11 in 2024-08-06T18:26:03.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:03.163 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:03.163 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:03.179 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:03.180 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:03.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:03.197 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:03.197 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:03.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:03.197 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:03.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:03.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:03.198 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:03.322 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:03.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:03.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:03.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:03.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:03.812 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:03.840 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:03.840 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:03.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:03.880 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:03.880 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:03.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:03 smithi107 ceph-mon[32381]: pgmap v1921: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:03.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1620176923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:03.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:03.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:03.881 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:03 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:03.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:04.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:03 smithi117 ceph-mon[32835]: pgmap v1921: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:04.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1620176923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:04.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:03 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:03 smithi086 ceph-mon[29229]: pgmap v1921: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1620176923' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:03 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:04.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:04.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:04.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:04.430 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:04.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:04.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:04.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:26:04.782 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:04.912 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:04.970 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:05.009 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:05.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:26:05.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:05.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:05.257 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:05.344 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:05.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:05.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:05.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:05.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:05.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:05.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:26:05.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:06.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:06.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:05 smithi117 ceph-mon[32835]: pgmap v1922: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:06.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:05 smithi086 ceph-mon[29229]: pgmap v1922: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:06.109 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:06.109 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:06.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:06.182 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:05 smithi107 ceph-mon[32381]: pgmap v1922: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:06.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:06.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:06.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:06.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:06.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:06.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:06.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:06.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:06.841 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:06.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:07.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:07.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:07.218 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:07.218 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:07.219 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:07.219 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:07.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:07.447 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:07.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:26:07.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:07.483 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:07.483 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:07.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:07.693 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:07.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:07.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:07.859 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:26:07.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:08.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:07 smithi086 ceph-mon[29229]: pgmap v1923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:08.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:07 smithi086 ceph-mon[29229]: osdmap e645: 12 total, 3 up, 11 in 2024-08-06T18:26:08.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:08.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:07 smithi117 ceph-mon[32835]: pgmap v1923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:08.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:07 smithi117 ceph-mon[32835]: osdmap e645: 12 total, 3 up, 11 in 2024-08-06T18:26:08.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:08.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:08.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:08.148 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:08.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:07 smithi107 ceph-mon[32381]: pgmap v1923: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:08.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:07 smithi107 ceph-mon[32381]: osdmap e645: 12 total, 3 up, 11 in 2024-08-06T18:26:08.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:08.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:08.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:08.280 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:08.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:08.460 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:08.460 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:08.460 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:08.477 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:08.477 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:08.477 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:08.477 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:08.478 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:08.495 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:08.495 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:08.495 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:08.495 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:08.496 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:08.496 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:08.496 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:08.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:08.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:08.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:08.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:08.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:08 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:08 smithi107 ceph-mon[32381]: osdmap e646: 12 total, 3 up, 11 in 2024-08-06T18:26:08.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1327872405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:09.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:09.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:08 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:09.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:09.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:08 smithi117 ceph-mon[32835]: osdmap e646: 12 total, 3 up, 11 in 2024-08-06T18:26:09.059 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1327872405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:09.073 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:08 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:09.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3204164358' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:09.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:08 smithi086 ceph-mon[29229]: osdmap e646: 12 total, 3 up, 11 in 2024-08-06T18:26:09.074 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1327872405' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:09.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:09.249 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:09.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:09.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:09.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:09.610 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:09.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:09.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:09.850 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:09 smithi117 ceph-mon[32835]: pgmap v1926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:09.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.851 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:09.860 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:09.860 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:09 smithi086 ceph-mon[29229]: pgmap v1926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:09.860 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.860 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.860 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:09.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:26:09.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:10.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:10.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:09 smithi107 ceph-mon[32381]: pgmap v1926: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:10.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:10.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:10.124 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:10.128 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:10.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:10.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:10.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:10.515 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:10.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:10.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:10.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:10.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:10.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:10.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:10.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:10.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:10.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:26:11.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:11.146 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:11.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:11.183 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:11.183 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:11.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:11.424 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:11.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:11.602 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:11.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:26:11.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:11.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:11.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:11 smithi086 ceph-mon[29229]: pgmap v1927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:11.849 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:11 smithi086 ceph-mon[29229]: osdmap e647: 12 total, 3 up, 11 in 2024-08-06T18:26:11.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:11.849 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:11.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:12.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:12.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:11 smithi117 ceph-mon[32835]: pgmap v1927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:12.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:11 smithi117 ceph-mon[32835]: osdmap e647: 12 total, 3 up, 11 in 2024-08-06T18:26:12.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:12.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:12.147 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:12.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:26:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:11 smithi107 ceph-mon[32381]: pgmap v1927: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:11 smithi107 ceph-mon[32381]: osdmap e647: 12 total, 3 up, 11 in 2024-08-06T18:26:12.221 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:12.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:12.433 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:12.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:12.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:12.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:12.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:12.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:12.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:12.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:13.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:13.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:13.175 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:13.175 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:13.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:13.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:13.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:13.683 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:13.704 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:13.704 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:13.755 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:26:13.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:13 smithi117 ceph-mon[32835]: pgmap v1929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:13 smithi117 ceph-mon[32835]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:13.923 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:13.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:13.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:13.960 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:13.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:13 smithi086 ceph-mon[29229]: pgmap v1929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:13.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:13 smithi086 ceph-mon[29229]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:13 smithi107 ceph-mon[32381]: pgmap v1929: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:13 smithi107 ceph-mon[32381]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:14.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:14.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:14.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:14.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:14.535 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:14.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:26:14.596 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:14.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:14.763 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:14 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:26:14.763 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:14 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:26:14.763 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:14 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:14.763 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:14 smithi117 ceph-mon[32835]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:14.764 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:14 smithi117 ceph-mon[32835]: osdmap e648: 12 total, 3 up, 11 in 2024-08-06T18:26:14.781 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:14.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:14 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:26:14.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:14 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:26:14.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:14 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:14.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:14 smithi086 ceph-mon[29229]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:14.816 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:14 smithi086 ceph-mon[29229]: osdmap e648: 12 total, 3 up, 11 in 2024-08-06T18:26:14.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:14 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:26:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:14 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:26:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:14 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:14 smithi107 ceph-mon[32381]: from='client.18885 172.21.15.86:0/884198485' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:14.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:14 smithi107 ceph-mon[32381]: osdmap e648: 12 total, 3 up, 11 in 2024-08-06T18:26:14.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:15.069 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:15.069 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:15.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:15.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:15.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:15.086 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:15.087 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:15.087 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:15.087 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:15.087 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:15.104 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:15.104 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:15.105 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:15.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:15.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:15.288 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:15.289 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:15.289 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:15.289 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:15.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:15.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:15.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:15.782 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:15.782 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:15.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:15.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: pgmap v1931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: osdmap e649: 12 total, 3 up, 11 in 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/580509502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:15.941 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:15.942 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:16.081 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: pgmap v1931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: osdmap e649: 12 total, 3 up, 11 in 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/580509502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: pgmap v1931: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:16.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:26:16.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:26:16.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: osdmap e649: 12 total, 3 up, 11 in 2024-08-06T18:26:16.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/580509502' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:16.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.125 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:16.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:16.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:16.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:16.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:26:16.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:16.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:16.743 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:16.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:26:16.818 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:16.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:16.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:16.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:16 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:26:16.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:16 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:26:16.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:16.942 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:16.994 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:17.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:17.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:17.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:16 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:26:17.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:16 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:26:17.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:16 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:26:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:16 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:26:17.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:17.396 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:17.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:17.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:17.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:26:17.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:17.806 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:17 smithi086 ceph-mon[29229]: pgmap v1933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:17.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:17.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:17.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:17.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:17.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:18.035 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:17 smithi117 ceph-mon[32835]: pgmap v1933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:18.043 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:18.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:18.189 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:17 smithi107 ceph-mon[32381]: pgmap v1933: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:18.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:18.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:18.481 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:18.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:18.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:18.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:18.752 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:18.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:18.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:19.063 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:19.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:26:19.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:19.145 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:19.145 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:19.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:19.313 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:26:19.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:19.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:19.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:19.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:19.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:19.787 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:19.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:19.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:19 smithi107 ceph-mon[32381]: pgmap v1934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:19.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:19 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:26:19.930 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:19 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:26:20.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:20.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:19 smithi086 ceph-mon[29229]: pgmap v1934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:20.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:19 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:26:20.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:19 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:26:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:19 smithi117 ceph-mon[32835]: pgmap v1934: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:19 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:26:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:19 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:26:20.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:20.150 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:20.164 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:20.164 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:20.165 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:20.165 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:20.165 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:20.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:20.181 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:20.182 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:20.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:20.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:20.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:20.183 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:20.183 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:20.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:20.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:20.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:20.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:20.563 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:20.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:20.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:20.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:20.751 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:20.751 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:20.802 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:26:20.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:20.871 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:20.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:20.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:20 smithi117 ceph-mon[32835]: osdmap e650: 12 total, 3 up, 11 in 2024-08-06T18:26:20.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2905270846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:20.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:20.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:20.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:21.087 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:21.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:20 smithi107 ceph-mon[32381]: osdmap e650: 12 total, 3 up, 11 in 2024-08-06T18:26:21.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2905270846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:21.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:21.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:21.109 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:20 smithi086 ceph-mon[29229]: osdmap e650: 12 total, 3 up, 11 in 2024-08-06T18:26:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2905270846' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:21.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:21.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:21.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:21.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:21.491 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:21.491 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:21.601 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:21.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:26:21.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:21.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:21.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:21.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:21.866 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:21.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:22.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:22.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:22.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:21 smithi107 ceph-mon[32381]: pgmap v1936: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:22.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:21 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:22.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:22.067 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:21 smithi107 ceph-mon[32381]: osdmap e651: 12 total, 3 up, 11 in 2024-08-06T18:26:22.081 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:22.081 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:22.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:22.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:21 smithi086 ceph-mon[29229]: pgmap v1936: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:22.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:21 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:22.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:22.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:21 smithi086 ceph-mon[29229]: osdmap e651: 12 total, 3 up, 11 in 2024-08-06T18:26:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:21 smithi117 ceph-mon[32835]: pgmap v1936: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:21 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1052996023' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:21 smithi117 ceph-mon[32835]: osdmap e651: 12 total, 3 up, 11 in 2024-08-06T18:26:22.176 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:22.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:22.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:22.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:22.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:22.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:26:22.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:22.965 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:23.026 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:23.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:23.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:23.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:23.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:23.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:26:23.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:23.682 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:23.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:23.917 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:23.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:23 smithi086 ceph-mon[29229]: pgmap v1938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:24.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:24.074 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:23 smithi107 ceph-mon[32381]: pgmap v1938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:24.075 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:24.075 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:23 smithi117 ceph-mon[32835]: pgmap v1938: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:24.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:24.266 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:24.266 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:24.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:24.454 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:24.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:24.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:24.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:24.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:24.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:24.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:25.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:25.255 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:25.258 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:25.265 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:25.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:25.267 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:25.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:25.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:25.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:25.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:25.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:25 smithi086 ceph-mon[29229]: pgmap v1939: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:25.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2572662419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:25.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:25.858 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:25.885 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:25.885 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:25.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:26.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:26.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:26.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:25 smithi117 ceph-mon[32835]: pgmap v1939: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2572662419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:26.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:25 smithi107 ceph-mon[32381]: pgmap v1939: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2572662419' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:26.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:26.387 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:26.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:26:26.415 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:26.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:26.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:26.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:26.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:26.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:26.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:26.885 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:26.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:27.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:27.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:27.149 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:27.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:27.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:27.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:27.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:27.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:27.811 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:27.811 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:27.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:26:27.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:27.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:27 smithi107 ceph-mon[32381]: pgmap v1940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:27.960 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:27.960 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:28.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:28.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:27 smithi086 ceph-mon[29229]: pgmap v1940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:27 smithi117 ceph-mon[32835]: pgmap v1940: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:28.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:28.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:28.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:28.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:28.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:28.605 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:28.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:28.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:29.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:29.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:29.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:29.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:29.434 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:29.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:29.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:29.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:29.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:29.819 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:29.829 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:26:29.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:30.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:30.102 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:29 smithi117 ceph-mon[32835]: pgmap v1941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:30.103 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2127951969' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.103 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:29 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:29 smithi086 ceph-mon[29229]: pgmap v1941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2127951969' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:29 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:29 smithi107 ceph-mon[32381]: pgmap v1941: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2127951969' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:29 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:30.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:30.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:30.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:30.672 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:30.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:30.780 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:30.817 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:30.920 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:30.946 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:30 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:30.946 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:30 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:30.950 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:30 smithi117 ceph-mon[32835]: osdmap e652: 12 total, 3 up, 11 in 2024-08-06T18:26:31.056 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:31.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:30 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:31.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:30 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:31.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:30 smithi086 ceph-mon[29229]: osdmap e652: 12 total, 3 up, 11 in 2024-08-06T18:26:31.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:31.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:30 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:31.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:30 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:31.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:30 smithi107 ceph-mon[32381]: osdmap e652: 12 total, 3 up, 11 in 2024-08-06T18:26:31.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:31.361 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:31.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:26:31.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:31.599 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:31.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:31.854 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:31.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:31.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:32.022 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:31 smithi086 ceph-mon[29229]: pgmap v1943: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:32.022 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:31 smithi086 ceph-mon[29229]: osdmap e653: 12 total, 3 up, 11 in 2024-08-06T18:26:32.022 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:32.022 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:31 smithi117 ceph-mon[32835]: pgmap v1943: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:31 smithi117 ceph-mon[32835]: osdmap e653: 12 total, 3 up, 11 in 2024-08-06T18:26:32.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:31 smithi107 ceph-mon[32381]: pgmap v1943: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:31 smithi107 ceph-mon[32381]: osdmap e653: 12 total, 3 up, 11 in 2024-08-06T18:26:32.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:32.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:32.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:32.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:32.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:32.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:26:32.665 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:32.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:32.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:32.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:32.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:33.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:33.063 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:33.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:33.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:33.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:33.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:33.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:33.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:33.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:33 smithi086 ceph-mon[29229]: pgmap v1945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:33 smithi117 ceph-mon[32835]: pgmap v1945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:34.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:34.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:33 smithi107 ceph-mon[32381]: pgmap v1945: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:34.151 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:34.152 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:34.252 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:34.252 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:34.252 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:34.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:34.269 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:34.269 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:34.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:34.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:34.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:34.291 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:34.291 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:34.292 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:34.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:34.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:34.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:34.531 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:34.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:34.782 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:34.799 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:34.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:35.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:35.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/686945452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:35.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:35.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/686945452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:35.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/686945452' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:35.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:35.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:35.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:35.602 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:35.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:35.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:35.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:35.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:35.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:35.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:35 smithi107 ceph-mon[32381]: pgmap v1946: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:35.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:36.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:36.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:35 smithi086 ceph-mon[29229]: pgmap v1946: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:36.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:36.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:35 smithi117 ceph-mon[32835]: pgmap v1946: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:36.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:36.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:36.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:36.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:36.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:36.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:36.788 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:36.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:26:36.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:36.835 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:36.835 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:36.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:37.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:37.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:37.346 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:37.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:37.505 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:37.505 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:37.541 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:37.541 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:37.572 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:37.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:37.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:37.841 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:26:37.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:38.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:37 smithi086 ceph-mon[29229]: pgmap v1947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:37 smithi117 ceph-mon[32835]: pgmap v1947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:38.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:37 smithi107 ceph-mon[32381]: pgmap v1947: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:26:38.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:38.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:38.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:38.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:38.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:38.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:38.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:38.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:38.708 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:38.708 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:38.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:38.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:38.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:38.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:38.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:38 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:38.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:38 smithi107 ceph-mon[32381]: osdmap e654: 12 total, 3 up, 11 in 2024-08-06T18:26:39.083 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:39.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:38 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:39.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:39.106 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:38 smithi086 ceph-mon[29229]: osdmap e654: 12 total, 3 up, 11 in 2024-08-06T18:26:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:38 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/740687380' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:26:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:38 smithi117 ceph-mon[32835]: osdmap e654: 12 total, 3 up, 11 in 2024-08-06T18:26:39.218 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:39.316 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:26:39.316 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:26:39.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:39.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:39.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:26:39.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:39.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:39.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:39.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:39.777 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:39.777 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:39.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:40.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:40.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:39 smithi086 ceph-mon[29229]: pgmap v1949: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:40.029 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:40.029 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:40.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:26:40.043 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:39 smithi117 ceph-mon[32835]: pgmap v1949: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:40.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:39 smithi107 ceph-mon[32381]: pgmap v1949: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:40.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:40.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:40.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:40.610 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:40.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:40.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:40.643 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:40.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:40.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:40.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:41.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:41.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:41.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:41.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:41.518 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:41.518 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:41.627 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:41.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:41.859 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:41.859 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:41.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:41.917 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:41 smithi086 ceph-mon[29229]: pgmap v1950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:41.917 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:41 smithi086 ceph-mon[29229]: osdmap e655: 12 total, 3 up, 11 in 2024-08-06T18:26:42.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:42.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:41 smithi117 ceph-mon[32835]: pgmap v1950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:42.052 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:41 smithi117 ceph-mon[32835]: osdmap e655: 12 total, 3 up, 11 in 2024-08-06T18:26:42.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:41 smithi107 ceph-mon[32381]: pgmap v1950: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:41 smithi107 ceph-mon[32381]: osdmap e655: 12 total, 3 up, 11 in 2024-08-06T18:26:42.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:42.422 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:42.451 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:42.451 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:42.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:26:42.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:42.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:42.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:42.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:42.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:42.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:43.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:43.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:43.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:43.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:43.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:43.690 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:26:43.690 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:26:43.691 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:26:43.691 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:26:43.691 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:26:43.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:43.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:43.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:43.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:44.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:43 smithi086 ceph-mon[29229]: pgmap v1952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/187221691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:43 smithi117 ceph-mon[32835]: pgmap v1952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/187221691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:44.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:43 smithi107 ceph-mon[32381]: pgmap v1952: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/187221691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:44.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:44.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:44.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:44.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:44.725 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:44.726 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:44.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:44.780 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:44.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:44.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:44.883 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:44.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:45.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:45.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:45.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:45.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:26:45.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:45.390 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:45.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:45.462 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:45.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:45.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:45.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:45.817 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:45.817 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:45.817 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:45.817 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:45.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:45.946 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:46.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:46.085 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:45 smithi107 ceph-mon[32381]: pgmap v1953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:45 smithi117 ceph-mon[32835]: pgmap v1953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:46.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:46.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:45 smithi086 ceph-mon[29229]: pgmap v1953: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:46.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:46.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:46.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:46.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:46.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:26:46.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:46.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:46.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:47.038 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:47.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:47.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:47.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:47.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:47.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:47.687 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:47.687 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:47.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:47.874 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:26:47.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:47.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:47 smithi086 ceph-mon[29229]: pgmap v1954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:47.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:47.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:47.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:48.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:48.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:48.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:48.212 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:47 smithi107 ceph-mon[32381]: pgmap v1954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:48.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:48.346 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:47 smithi117 ceph-mon[32835]: pgmap v1954: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:48.346 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:26:48.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:48.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:48.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:48.748 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:48.749 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:48.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:48.978 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:49.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:48 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:49.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:49.000 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:48 smithi086 ceph-mon[29229]: osdmap e656: 12 total, 3 up, 11 in 2024-08-06T18:26:49.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:49.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:48 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:49.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:49.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:48 smithi107 ceph-mon[32381]: osdmap e656: 12 total, 3 up, 11 in 2024-08-06T18:26:49.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:49.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:48 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:26:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/111163406' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:26:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:48 smithi117 ceph-mon[32835]: osdmap e656: 12 total, 3 up, 11 in 2024-08-06T18:26:49.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:49.390 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:49.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:49.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:49.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:49.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:49.926 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:49 smithi117 ceph-mon[32835]: pgmap v1956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:50.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:50.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:50.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:49 smithi086 ceph-mon[29229]: pgmap v1956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:49 smithi107 ceph-mon[32381]: pgmap v1956: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:50.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:50.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:50.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:50.685 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:50.686 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:50.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:50.687 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:50.703 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:50.704 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:50.704 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:50.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:50.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:50.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:50.705 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:50.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:26:50.709 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:50.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:50.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:50.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3144748842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:50.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:50 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:26:50.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:50 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:26:50.999 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:51.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3144748842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:50 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:26:51.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:50 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:26:51.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:51.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:51.260 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:51.260 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:51.347 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3144748842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:51.347 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:50 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:26:51.347 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:50 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:26:51.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:51.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:51.403 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:51.404 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:51.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:51.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:51.516 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:51.613 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:26:51.614 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:26:51.630 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:51.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:51.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:51.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:51.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:51 smithi107 ceph-mon[32381]: pgmap v1957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:51 smithi107 ceph-mon[32381]: osdmap e657: 12 total, 3 up, 11 in 2024-08-06T18:26:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:51 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:26:52.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:51 smithi117 ceph-mon[32835]: pgmap v1957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:51 smithi117 ceph-mon[32835]: osdmap e657: 12 total, 3 up, 11 in 2024-08-06T18:26:52.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:51 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:26:52.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:52.180 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:52.180 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:52.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:51 smithi086 ceph-mon[29229]: pgmap v1957: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:52.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:52.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:52.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:51 smithi086 ceph-mon[29229]: osdmap e657: 12 total, 3 up, 11 in 2024-08-06T18:26:52.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:51 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:26:52.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:52.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:26:52.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:52.435 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:52.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:52.557 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:52.557 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:52.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:52.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:52.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:26:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:26:52.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:26:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:52.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:53.074 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:53.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:26:53.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:26:53.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:26:53.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:26:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:26:53.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:26:53.204 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:53.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:53.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:26:53.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:53.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:53.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:53.734 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:53.863 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:53.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:54.021 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:53 smithi107 ceph-mon[32381]: pgmap v1959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:54.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:53 smithi086 ceph-mon[29229]: pgmap v1959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:54.130 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:54.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:53 smithi117 ceph-mon[32835]: pgmap v1959: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:54.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:54.449 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:54.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:26:54.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:54.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:54.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:54.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:54.903 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:26:54.903 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:26:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:26:54.908 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:26:54.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:54.978 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:55.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:55.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:26:55.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:55.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:55.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:55.621 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:55.757 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:55.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:56.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:56.046 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:56.046 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:55 smithi086 ceph-mon[29229]: pgmap v1960: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:55 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:56.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:55 smithi086 ceph-mon[29229]: osdmap e658: 12 total, 3 up, 11 in 2024-08-06T18:26:56.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:56.181 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:55 smithi107 ceph-mon[32381]: pgmap v1960: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:56.181 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:55 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:56.181 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:56.181 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:55 smithi107 ceph-mon[32381]: osdmap e658: 12 total, 3 up, 11 in 2024-08-06T18:26:56.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:55 smithi117 ceph-mon[32835]: pgmap v1960: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:55 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:26:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3852214623' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:26:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:55 smithi117 ceph-mon[32835]: osdmap e658: 12 total, 3 up, 11 in 2024-08-06T18:26:56.423 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:26:56.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:56.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:56.758 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:26:56.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:56.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:57.099 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:57.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:57.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:26:57.454 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:26:57.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:26:57.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:57.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:57.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:57.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:26:57.545 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:26:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:26:57.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:57.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:57.794 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:57 smithi117 ceph-mon[32835]: pgmap v1962: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:57.794 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:57 smithi117 ceph-mon[32835]: osdmap e659: 12 total, 3 up, 11 in 2024-08-06T18:26:57.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:57 smithi086 ceph-mon[29229]: pgmap v1962: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:57.846 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:57 smithi086 ceph-mon[29229]: osdmap e659: 12 total, 3 up, 11 in 2024-08-06T18:26:57.902 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:26:58.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:58.167 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:58.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:57 smithi107 ceph-mon[32381]: pgmap v1962: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:58.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:57 smithi107 ceph-mon[32381]: osdmap e659: 12 total, 3 up, 11 in 2024-08-06T18:26:58.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:58.374 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:26:58.375 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:26:58.375 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:26:58.388 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:26:58.389 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:26:58.389 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:26:58.389 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:26:58.389 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:26:58.406 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:26:58.407 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:26:58.407 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:26:58.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:26:58.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:26:58.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:26:58.408 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:26:58.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:58.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:26:58.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:26:58.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:58.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:26:59.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:26:59.003 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:26:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:26:59.003 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2015960334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:59.030 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:26:59.031 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:26:59.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:26:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2015960334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2015960334' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:26:59.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:59.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:26:59.502 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:26:59.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:26:59.699 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:26:59.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:26:59.791 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:59 smithi107 ceph-mon[32381]: pgmap v1964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:59.791 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.791 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.791 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:26:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:59 smithi086 ceph-mon[29229]: pgmap v1964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:26:59.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:26:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:26:59.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:00.031 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:59 smithi117 ceph-mon[32835]: pgmap v1964: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:26:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:00.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:00.184 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:00.184 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:00.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:00.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:00.471 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:00.471 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:00.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:00.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:00.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:00.780 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:00.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:00.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:01.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:27:01.054 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:01.054 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:01.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:01.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:01.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:01.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:01.299 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:01.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:01.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:01.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:01.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:01.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:27:01.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:01.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:01.837 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:01 smithi086 ceph-mon[29229]: pgmap v1965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:01.837 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:01.837 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:01.838 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:01.838 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:01.946 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:02.064 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:02.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:27:02.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:02.107 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:01 smithi107 ceph-mon[32381]: pgmap v1965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:02.107 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:02.107 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:01 smithi117 ceph-mon[32835]: pgmap v1965: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:02.131 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:02.140 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:02.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:02.351 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:02.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:02.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:02.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:02.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:02.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:03.025 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:03.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:03.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:03.416 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:03.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:03.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:03.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:03.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:03.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:03 smithi117 ceph-mon[32835]: pgmap v1966: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:03.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:04.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:04.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:03 smithi086 ceph-mon[29229]: pgmap v1966: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:04.105 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:04.105 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:03 smithi107 ceph-mon[32381]: pgmap v1966: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:04.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:04.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:04.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:04.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:04.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:04.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:04.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:04.632 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:04.763 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:04.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:04.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:04.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:04.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:04.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:05.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:05.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:05.051 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:05.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:05.186 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:05.186 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:05.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:05.430 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:05.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:05.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:05.766 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:27:05.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:05.808 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:05.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:05.825 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:05.826 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:05.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:05.843 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:05.844 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:05.844 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:05.844 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:05.844 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:05.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:05.845 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:05.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:05.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:05 smithi086 ceph-mon[29229]: pgmap v1967: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:05.976 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:06.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:06.118 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:05 smithi117 ceph-mon[32835]: pgmap v1967: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:06.118 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:05 smithi107 ceph-mon[32381]: pgmap v1967: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:06.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:06.250 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:06.251 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:06.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:06.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:06.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:06.679 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:06.680 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:06.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:06.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:06.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:06.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:07.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:07.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: osdmap e660: 12 total, 3 up, 11 in 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2603178319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:06 smithi086 ceph-mon[29229]: osdmap e661: 12 total, 3 up, 11 in 2024-08-06T18:27:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: osdmap e660: 12 total, 3 up, 11 in 2024-08-06T18:27:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2603178319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:06 smithi117 ceph-mon[32835]: osdmap e661: 12 total, 3 up, 11 in 2024-08-06T18:27:07.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:07.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:07.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3104614497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:07.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: osdmap e660: 12 total, 3 up, 11 in 2024-08-06T18:27:07.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2603178319' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:07.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:07.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:06 smithi107 ceph-mon[32381]: osdmap e661: 12 total, 3 up, 11 in 2024-08-06T18:27:07.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:07.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:07.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:07.588 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:07.588 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:07.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:07.787 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:07.787 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:07.787 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:07.787 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:07.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:07.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:07 smithi086 ceph-mon[29229]: pgmap v1969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:07.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:08.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:07 smithi117 ceph-mon[32835]: pgmap v1969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:08.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:07 smithi107 ceph-mon[32381]: pgmap v1969: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:08.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:08.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:08.468 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:08.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:08.643 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:08.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:27:08.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:08.788 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:08.788 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:08.916 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:08.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:08.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:09.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:09.174 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:09.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:27:09.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:09.267 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:09.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:09.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:09.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:27:09.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:09.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:09.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:09 smithi107 ceph-mon[32381]: pgmap v1971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:09.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:10.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:09 smithi086 ceph-mon[29229]: pgmap v1971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:10.016 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:10.016 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:09 smithi117 ceph-mon[32835]: pgmap v1971: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:10.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:10.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:10.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:10.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:10.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:10.682 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:10.682 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:10.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:10.928 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:10.950 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:10.950 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:11.035 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:11.035 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:11.040 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:11.047 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:11.047 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:11.047 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:11.048 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:11.050 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:11.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:11.068 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:11.069 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:11.070 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:11.071 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:11.075 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:11.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:11.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:11.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:11.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:11.715 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:11.715 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:11.732 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:11.792 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:11.792 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:11.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:12.003 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:12.003 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:12.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:12.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:11 smithi117 ceph-mon[32835]: pgmap v1972: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:12.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1207200797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:12.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.031 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:11 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:11 smithi086 ceph-mon[29229]: pgmap v1972: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1207200797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:11 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:11 smithi107 ceph-mon[32381]: pgmap v1972: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1207200797' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:11 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:12.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:12.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:12.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:12.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:12.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:12.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:12.798 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:12.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:12.957 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:13.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:13.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:13.332 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:13.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:13.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:13.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:13.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:13.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:13.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:13.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:14.004 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:27:14.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:14.026 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:13 smithi086 ceph-mon[29229]: pgmap v1973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:14.026 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:14.027 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:13 smithi117 ceph-mon[32835]: pgmap v1973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:14.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:13 smithi107 ceph-mon[32381]: pgmap v1973: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:14.275 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:14.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:14.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:14.566 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:14.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:14.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:14.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:14.804 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:27:14.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:14.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:14.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:14.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:14.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:14.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:14.970 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:15.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:15.209 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:15.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:15.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:15.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:15.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:15.612 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:15.742 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:15.774 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:15.775 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:15.878 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:16.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:16.030 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:16.030 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:16.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:15 smithi086 ceph-mon[29229]: pgmap v1974: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:16.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:15 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:16.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:16.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:15 smithi086 ceph-mon[29229]: osdmap e662: 12 total, 3 up, 11 in 2024-08-06T18:27:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:15 smithi117 ceph-mon[32835]: pgmap v1974: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:15 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:15 smithi117 ceph-mon[32835]: osdmap e662: 12 total, 3 up, 11 in 2024-08-06T18:27:16.158 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:15 smithi107 ceph-mon[32381]: pgmap v1974: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:15 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2191957175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:15 smithi107 ceph-mon[32381]: osdmap e662: 12 total, 3 up, 11 in 2024-08-06T18:27:16.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:16.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:16.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:16.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:16.713 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:27:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:27:16.713 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:27:16.713 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:27:16.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:16.961 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:17.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:17.227 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:17.358 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:17.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:17.510 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:17.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:17.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:17.755 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:17.789 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:17.790 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:17.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:17 smithi086 ceph-mon[29229]: pgmap v1976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:17.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:17 smithi086 ceph-mon[29229]: osdmap e663: 12 total, 3 up, 11 in 2024-08-06T18:27:17.790 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4025159530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:17.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:27:17.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:27:17.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:17.923 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:17.923 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:17.924 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:17.925 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:17.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:17.925 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:17.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:17.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:17.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:17.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:18.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:17 smithi117 ceph-mon[32835]: pgmap v1976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:17 smithi117 ceph-mon[32835]: osdmap e663: 12 total, 3 up, 11 in 2024-08-06T18:27:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4025159530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:18.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:18.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:17 smithi107 ceph-mon[32381]: pgmap v1976: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:17 smithi107 ceph-mon[32381]: osdmap e663: 12 total, 3 up, 11 in 2024-08-06T18:27:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4025159530' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:18.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:18.319 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:18.319 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:18.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:18.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:18.589 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:18.589 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:18.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:18.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:18.688 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:18.689 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:18.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:18.773 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:18.776 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:18.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:18.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:18.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:18.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:19.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:19.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:19.258 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:19.258 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:19.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:19.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:19.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:19.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:19.844 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:19 smithi086 ceph-mon[29229]: pgmap v1978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:19.845 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:19.845 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:19.922 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:20.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:20.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:19 smithi107 ceph-mon[32381]: pgmap v1978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:19 smithi117 ceph-mon[32835]: pgmap v1978: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 330 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:20.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:20.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:20.352 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:20.352 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:20.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:20.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:20.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:27:20.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:20.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:20.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:20.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:20.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:21.019 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:21.019 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:21.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:21.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:21.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:21.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:21.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:21.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:21.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:21 smithi086 ceph-mon[29229]: pgmap v1979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:21.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:22.044 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:22.050 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:22.057 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:27:22.058 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:27:22.070 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:21 smithi107 ceph-mon[32381]: pgmap v1979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:21 smithi117 ceph-mon[32835]: pgmap v1979: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:22.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:22.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:22.352 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:22.352 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:22.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:22.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:22.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:22.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:22.774 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:27:22.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:22.859 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:22.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:22.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:23.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:23.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:23.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:23.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:23.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:23.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:23.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:23.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:23.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:23.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:23.893 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:24.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:24.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:23 smithi086 ceph-mon[29229]: pgmap v1980: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:24.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:23 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:24.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:24.036 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:23 smithi086 ceph-mon[29229]: osdmap e664: 12 total, 3 up, 11 in 2024-08-06T18:27:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:23 smithi117 ceph-mon[32835]: pgmap v1980: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:23 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:23 smithi117 ceph-mon[32835]: osdmap e664: 12 total, 3 up, 11 in 2024-08-06T18:27:24.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:24.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:23 smithi107 ceph-mon[32381]: pgmap v1980: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:24.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:23 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:24.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3883656388' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:24.184 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:23 smithi107 ceph-mon[32381]: osdmap e664: 12 total, 3 up, 11 in 2024-08-06T18:27:24.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:27:24.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:24.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:24.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:24.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:24.758 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:24.759 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:24.759 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:24.767 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:24.777 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:24.777 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:24.778 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:24.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:24.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:24.798 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:24.799 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:24.799 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3473260590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:24.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:24.834 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:24.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:24.994 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3473260590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:25.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:25.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:25.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3473260590' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:25.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:25.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:25.503 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:25.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:25.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:25.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:25.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:25.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:25.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:27:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:25 smithi107 ceph-mon[32381]: pgmap v1982: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:25.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:26.059 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:26.059 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:26.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:25 smithi086 ceph-mon[29229]: pgmap v1982: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:26.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.060 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:25 smithi117 ceph-mon[32835]: pgmap v1982: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:26.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:26.282 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:26.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:26.438 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:26.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:26.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:26.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:26.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:26.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:26.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:27.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:27.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:27.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:27.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:27.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:27.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:27.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:27:27.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:27.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:27.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:27.778 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:27 smithi117 ceph-mon[32835]: pgmap v1983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:27.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:27 smithi117 ceph-mon[32835]: osdmap e665: 12 total, 3 up, 11 in 2024-08-06T18:27:27.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:27 smithi107 ceph-mon[32381]: pgmap v1983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:27.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:27 smithi107 ceph-mon[32381]: osdmap e665: 12 total, 3 up, 11 in 2024-08-06T18:27:27.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:27.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:28.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:28.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:27 smithi086 ceph-mon[29229]: pgmap v1983: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:28.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:27 smithi086 ceph-mon[29229]: osdmap e665: 12 total, 3 up, 11 in 2024-08-06T18:27:28.065 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:28.065 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:28.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:28.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:28.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:28.573 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:28.598 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:28.598 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:28.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:28.775 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:28.775 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:28.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:28.869 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:28.870 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:27:28.875 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 426.48913073300002, 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017163700000000001, 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:27:28.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387807+0000", 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0500000000000002e-07 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387810+0000", 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3239999999999999e-06 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387819+0000", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387824+0000", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3630000000000002e-06 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387861+0000", 2024-08-06T18:27:28.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6433000000000002e-05 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387875+0000", 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4549000000000001e-05 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387978+0000", 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010249 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:27:28.879 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 426.48849134699998, 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2353000000000003e-05, 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:27:28.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4600000000000005e-07 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388449+0000", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.174e-06 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388457+0000", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1240000000000005e-06 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388462+0000", 2024-08-06T18:27:28.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4399999999999998e-06 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388493+0000", 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1256e-05 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388500+0000", 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4690000000000002e-06 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:27:28.883 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388518+0000", 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8343999999999999e-05 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:27:28.884 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:28.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:29.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:29.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:29.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:29.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:29.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:29.762 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:29.763 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:29.769 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:27:29.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:29.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:30.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:29 smithi086 ceph-mon[29229]: pgmap v1985: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:30.058 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:30.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:30.083 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:29 smithi107 ceph-mon[32381]: pgmap v1985: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:30.083 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:30.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:30.083 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:29 smithi117 ceph-mon[32835]: pgmap v1985: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:30.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:30.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:30.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:30.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:30.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:30.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:30.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:30.887 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:30 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:30.887 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:30.887 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:30 smithi086 ceph-mon[29229]: osdmap e666: 12 total, 3 up, 11 in 2024-08-06T18:27:30.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:27:30.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:30.942 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:30.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:30 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:30.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:30.943 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:30 smithi107 ceph-mon[32381]: osdmap e666: 12 total, 3 up, 11 in 2024-08-06T18:27:31.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:31.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:30 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:31.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2920244060' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:31.028 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:30 smithi117 ceph-mon[32835]: osdmap e666: 12 total, 3 up, 11 in 2024-08-06T18:27:31.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:31.175 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:31.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:31.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:31.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:31.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:31.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:31.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:31.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:31.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:31 smithi107 ceph-mon[32381]: pgmap v1987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:31.817 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:31 smithi107 ceph-mon[32381]: osdmap e667: 12 total, 3 up, 11 in 2024-08-06T18:27:31.925 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:31.926 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:31.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:31.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:31.942 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:31.942 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:31.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:31 smithi086 ceph-mon[29229]: pgmap v1987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:31.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:31 smithi086 ceph-mon[29229]: osdmap e667: 12 total, 3 up, 11 in 2024-08-06T18:27:31.943 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:31.943 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:31.944 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:31.959 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:31.960 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:31.960 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:31.960 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:31.960 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:31.961 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:31.961 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:32.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:32.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:31 smithi117 ceph-mon[32835]: pgmap v1987: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:32.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:31 smithi117 ceph-mon[32835]: osdmap e667: 12 total, 3 up, 11 in 2024-08-06T18:27:32.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:32.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:32.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:32.550 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:27:32.578 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:32.579 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:32.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:32.727 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:32.801 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:32.857 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:32 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:27:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:32 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:27:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4078368627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:32.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:32.924 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:32 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:27:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:32 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:27:32.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4078368627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:32.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:32.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:33.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:32 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:27:33.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:32 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:27:33.047 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4078368627' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:33.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:33.048 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:33.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:33.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:33.333 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:33.333 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:33.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:33.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:33.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:27:33.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:33.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:33.731 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:33.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:33.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:33.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:33.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:33.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:33 smithi107 ceph-mon[32381]: pgmap v1989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:34.050 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:34.076 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:34.076 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:33 smithi117 ceph-mon[32835]: pgmap v1989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:34.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:33 smithi086 ceph-mon[29229]: pgmap v1989: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:34.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:34.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:34.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:34.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:34.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:27:34.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:34.754 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:34.786 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:27:34.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:35.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:35.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:35.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:35.119 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:35.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:35.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:35.343 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:35.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:35.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:35.567 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:35.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:35.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:35.780 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:35.821 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:35.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:35 smithi107 ceph-mon[32381]: pgmap v1990: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:35 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:35.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:35 smithi107 ceph-mon[32381]: osdmap e668: 12 total, 3 up, 11 in 2024-08-06T18:27:36.016 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:36.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:27:36.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:36.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:35 smithi117 ceph-mon[32835]: pgmap v1990: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:36.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:35 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:36.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:36.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:35 smithi117 ceph-mon[32835]: osdmap e668: 12 total, 3 up, 11 in 2024-08-06T18:27:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:35 smithi086 ceph-mon[29229]: pgmap v1990: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:35 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1582808739' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:35 smithi086 ceph-mon[29229]: osdmap e668: 12 total, 3 up, 11 in 2024-08-06T18:27:36.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:36.307 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:36.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:36.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:36.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:36.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:36.617 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:36.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:36.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:37.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:37.144 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:37.236 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:37.236 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:37.252 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:37.253 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:37.270 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:37.271 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:37.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:37.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:37.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:37.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:37.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:27:37.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:37.674 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:37.675 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:37.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:37.759 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:37.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:37 smithi086 ceph-mon[29229]: pgmap v1992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:37 smithi086 ceph-mon[29229]: osdmap e669: 12 total, 3 up, 11 in 2024-08-06T18:27:37.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1476471038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:37.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:37.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:37.900 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:37.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:37.932 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:37.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:37 smithi107 ceph-mon[32381]: pgmap v1992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:37 smithi107 ceph-mon[32381]: osdmap e669: 12 total, 3 up, 11 in 2024-08-06T18:27:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1476471038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:38.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:38.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:37 smithi117 ceph-mon[32835]: pgmap v1992: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:38.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:37 smithi117 ceph-mon[32835]: osdmap e669: 12 total, 3 up, 11 in 2024-08-06T18:27:38.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1476471038' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:38.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:38.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:38.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:38.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:38.379 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:38.379 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:27:38.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:38.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:38.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:38.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:38.624 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:38.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:38.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:38.899 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:39.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:39.034 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:39.072 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:39.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:39.165 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:39.165 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:39.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:39.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:39.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:39.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:39.692 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:39.692 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:39.770 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:27:39.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:39.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:27:39.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:39.960 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:39.960 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:39.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:39 smithi107 ceph-mon[32381]: pgmap v1994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:39.961 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:40.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:40.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:39 smithi086 ceph-mon[29229]: pgmap v1994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:40.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:39 smithi117 ceph-mon[32835]: pgmap v1994: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:27:40.154 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:40.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:40.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:40.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:40.583 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:40.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:40.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:40.859 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:40.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:27:40.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:41.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:41.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:41.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:40 smithi117 ceph-mon[32835]: osdmap e670: 12 total, 3 up, 11 in 2024-08-06T18:27:41.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:41.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:40 smithi086 ceph-mon[29229]: osdmap e670: 12 total, 3 up, 11 in 2024-08-06T18:27:41.109 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:41.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:41.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2173598820' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:27:41.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:40 smithi107 ceph-mon[32381]: osdmap e670: 12 total, 3 up, 11 in 2024-08-06T18:27:41.189 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:41.189 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:41.298 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:41.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:41.459 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:41.459 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:41.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:41.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:41.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:41.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:41.974 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:41 smithi086 ceph-mon[29229]: pgmap v1996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:41.974 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:41 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:41.975 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:41 smithi086 ceph-mon[29229]: osdmap e671: 12 total, 3 up, 11 in 2024-08-06T18:27:42.082 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:42.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:41 smithi117 ceph-mon[32835]: pgmap v1996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:42.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:41 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:42.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:41 smithi117 ceph-mon[32835]: osdmap e671: 12 total, 3 up, 11 in 2024-08-06T18:27:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:41 smithi107 ceph-mon[32381]: pgmap v1996: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:41 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:42.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:41 smithi107 ceph-mon[32381]: osdmap e671: 12 total, 3 up, 11 in 2024-08-06T18:27:42.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:42.335 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:42.335 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:42.335 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:42.335 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:42.336 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:42.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:42.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:42.336 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:42.336 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:42.337 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:42.337 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:42.337 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:42.337 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:42.337 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:42.338 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:42.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:42.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:27:42.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:42.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:42.500 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:42.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:42.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:42.819 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:42.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:42.898 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3783465927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:42.898 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:43.033 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3783465927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:43.033 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3783465927' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:43.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:43.067 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:43.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:43.164 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:43.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:43.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:43.291 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:43.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:43.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:27:43.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:43.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:43.633 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:43.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:43.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:43.832 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:43 smithi086 ceph-mon[29229]: pgmap v1998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:43 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:27:43.833 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:43 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:27:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:43 smithi117 ceph-mon[32835]: pgmap v1998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:43 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:27:43.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:43 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:27:43.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:43.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:43 smithi107 ceph-mon[32381]: pgmap v1998: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:43.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:43.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:43 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:27:43.952 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:43 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:27:44.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:44.097 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:44.098 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:44.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:44.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:44.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:44.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:44.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:44.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:44.633 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:44.633 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:44.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:44.759 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:44.759 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:44.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:44.894 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:44 smithi107 ceph-mon[32381]: osdmap e672: 12 total, 3 up, 11 in 2024-08-06T18:27:45.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:45.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:44 smithi086 ceph-mon[29229]: osdmap e672: 12 total, 3 up, 11 in 2024-08-06T18:27:45.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:27:45.112 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:44 smithi117 ceph-mon[32835]: osdmap e672: 12 total, 3 up, 11 in 2024-08-06T18:27:45.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:45.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:45.333 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:45.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:45.543 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:45.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:45.722 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:45.722 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:45.758 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:45.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:27:45.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:45.811 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:27:45.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:46.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:45 smithi086 ceph-mon[29229]: pgmap v2000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:46.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:46.010 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:46.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:46.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:45 smithi117 ceph-mon[32835]: pgmap v2000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:46.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:46.132 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:45 smithi107 ceph-mon[32381]: pgmap v2000: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:46.132 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:46.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:46.345 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:46.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:46.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:46.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:46.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:46.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:46.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:46.806 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:46.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:47.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:47.030 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:46 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:47.030 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:47.030 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:46 smithi117 ceph-mon[32835]: osdmap e673: 12 total, 3 up, 11 in 2024-08-06T18:27:47.095 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:46 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:47.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:47.096 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:46 smithi107 ceph-mon[32381]: osdmap e673: 12 total, 3 up, 11 in 2024-08-06T18:27:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:46 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:27:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1872288175' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:46 smithi086 ceph-mon[29229]: osdmap e673: 12 total, 3 up, 11 in 2024-08-06T18:27:47.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:47.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:47.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:47.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:47.642 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:27:47.642 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:27:47.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:47.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:47.909 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:47.909 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:47.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:47.926 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:47.926 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:47.926 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:47.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:47.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:47.934 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:47.943 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:47.944 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:47.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:47.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:47.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:47 smithi086 ceph-mon[29229]: pgmap v2002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:47.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:47 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:27:47.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:47 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:27:47.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:47.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:47.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:47.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:47.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:47 smithi107 ceph-mon[32381]: pgmap v2002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:47 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:27:47.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:47 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:27:48.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:48.092 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:47 smithi117 ceph-mon[32835]: pgmap v2002: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:48.092 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:47 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:27:48.092 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:47 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:27:48.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:48.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:27:48.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:48.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:48.367 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:48.457 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:48.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:48.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:48.686 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:48.687 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:48.732 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:48.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:48.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:48.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:48.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1924774442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:48.867 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:48.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:27:48.997 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:49.116 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1924774442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:49.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1924774442' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:49.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:49.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:49.475 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:49.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:49.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:49.476 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:49.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:49.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:49.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:49.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:49.974 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:27:49.974 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:27:49.974 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:27:49.974 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:27:49.974 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:27:49.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:49.996 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:49 smithi107 ceph-mon[32381]: pgmap v2003: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:50.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:49 smithi086 ceph-mon[29229]: pgmap v2003: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:49 smithi117 ceph-mon[32835]: pgmap v2003: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:50.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:50.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:50.348 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:50.348 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:50.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:50.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:50.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:50.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:50.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:27:50.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:27:50.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:50.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:50.889 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:50.890 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:50.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:50 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:27:50.890 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:50 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:27:50.945 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:50.968 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:51.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:51.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:50 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:27:51.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:50 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:27:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:50 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:27:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:50 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:27:51.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:51.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:51.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:51.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:51.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:51.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:51.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:51.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:27:51.828 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:27:51.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:51.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:51 smithi107 ceph-mon[32381]: pgmap v2004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:51.957 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:52.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:52.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:51 smithi086 ceph-mon[29229]: pgmap v2004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:52.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:51 smithi117 ceph-mon[32835]: pgmap v2004: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:27:52.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:52.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:52.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 447.178404683, 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000266113, 2024-08-06T18:27:52.582 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.583 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404599+0000", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.023e-06 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404602+0000", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.534e-06 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404616+0000", 2024-08-06T18:27:52.584 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3881e-05 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404626+0000", 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0402e-05 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404697+0000", 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0785999999999996e-05 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.585 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404716+0000", 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9258000000000001e-05 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404864+0000", 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147229 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:27:52.586 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 447.17763010300001, 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014078100000000001, 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.587 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405373+0000", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4000000000000001e-07 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405377+0000", 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.391e-06 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.588 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405387+0000", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0040999999999999e-05 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405393+0000", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.976e-06 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405481+0000", 2024-08-06T18:27:52.589 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7995e-05 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405490+0000", 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1649999999999995e-06 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405513+0000", 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2572999999999999e-05 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.590 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:575,from:0,reservation_nonce:0)", 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 365.00498120200001, 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039962, 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:27:52.591 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578023+0000", 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.736e-06 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.592 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578030+0000", 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0140000000000001e-06 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578056+0000", 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6302999999999999e-05 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578093+0000", 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6529000000000002e-05 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.593 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578174+0000", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1209000000000001e-05 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578203+0000", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9045999999999999e-05 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578421+0000", 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021778300000000001 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.594 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 365.00356723800002, 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030426500000000002, 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.595 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579437+0000", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.734e-06 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.596 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579490+0000", 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3355e-05 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579515+0000", 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5236999999999999e-05 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579527+0000", 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1482e-05 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.597 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579637+0000", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011032200000000001 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579666+0000", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9186000000000001e-05 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579739+0000", 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2948999999999996e-05 2024-08-06T18:27:52.598 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:599,from:0,reservation_nonce:0)", 2024-08-06T18:27:52.599 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.80063210200001, 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188605, 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.602 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782371+0000", 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8299999999999995e-07 2024-08-06T18:27:52.603 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782375+0000", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7780000000000001e-06 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782384+0000", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.986e-06 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782392+0000", 2024-08-06T18:27:52.604 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6650000000000003e-06 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782430+0000", 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8778e-05 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782444+0000", 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3981999999999999e-05 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.605 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782559+0000", 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011443299999999999 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 264.79998382100001, 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5184000000000005e-05, 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.606 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.607 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783019+0000", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7199999999999999e-07 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783022+0000", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4340000000000001e-06 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783029+0000", 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6590000000000001e-06 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.608 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783032+0000", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2650000000000001e-06 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783061+0000", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8751999999999998e-05 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783067+0000", 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-08-06T18:27:52.609 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783084+0000", 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:649,from:0,reservation_nonce:0)", 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.483959686000006, 2024-08-06T18:27:52.610 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021522199999999999, 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.611 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099044+0000", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.719e-06 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099049+0000", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4410000000000003e-06 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099061+0000", 2024-08-06T18:27:52.612 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2422999999999999e-05 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099070+0000", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6349999999999995e-06 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099113+0000", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3393999999999998e-05 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099128+0000", 2024-08-06T18:27:52.613 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5233e-05 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099258+0000", 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012937699999999999 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 96.483122885, 2024-08-06T18:27:52.614 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4050999999999999e-05, 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:27:52.615 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099880+0000", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5199999999999996e-07 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099886+0000", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1040000000000003e-06 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099896+0000", 2024-08-06T18:27:52.616 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8789999999999998e-06 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099901+0000", 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970000000000003e-06 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099938+0000", 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6542999999999999e-05 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:27:52.617 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099944+0000", 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099963+0000", 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998e-05 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:27:52.618 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:27:52.619 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:52.619 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:52.619 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:52.619 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:52.619 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:52.754 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:52.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:53.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:53.027 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:53.027 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: osdmap e674: 12 total, 3 up, 11 in 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:27:53.028 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:27:53.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:52 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:27:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: osdmap e674: 12 total, 3 up, 11 in 2024-08-06T18:27:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:27:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:27:53.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:52 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:27:53.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1976526586' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:27:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: osdmap e674: 12 total, 3 up, 11 in 2024-08-06T18:27:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:27:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:27:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:52 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:27:53.248 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:53.248 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:53.249 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:53.265 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:53.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:53.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:53.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:53.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:53.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:53.285 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:53.285 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:53.286 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:53.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:53.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:27:53.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:53.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:53.618 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:53.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:53.805 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:53.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:53.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/865799765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:53.960 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/865799765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:54.047 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:54.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:54.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/865799765' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:54.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:54.250 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:54.250 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:54.250 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:54.250 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:54.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:54.466 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:54.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:54.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:54.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:27:54.864 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:54.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:54.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:54.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:55.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:55.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:55.097 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:55.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:55.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:55.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:55.153 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:55.154 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:55.176 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:55.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:55.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:55.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:55.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:55.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:27:55.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:55.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:55.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:55.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:55.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:55.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:56.035 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:56.049 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:27:56.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:56.157 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:56.157 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: pgmap v2007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:56.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:56.247 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: pgmap v2007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.322 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:27:56.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: pgmap v2007: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:27:56.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:27:56.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:27:56.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:27:56.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:56.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:27:56.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:56.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:27:56.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:27:56.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:27:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:27:57.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:57.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:57.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:57.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:57.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:57.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:57.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:27:57.552 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:27:57.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:57.768 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:57.768 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:57.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:27:57.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:57 smithi117 ceph-mon[32835]: pgmap v2008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:57.840 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:57 smithi117 ceph-mon[32835]: osdmap e675: 12 total, 3 up, 11 in 2024-08-06T18:27:57.841 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:57.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:27:57.964 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:27:58.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:57 smithi086 ceph-mon[29229]: pgmap v2008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:58.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:57 smithi086 ceph-mon[29229]: osdmap e675: 12 total, 3 up, 11 in 2024-08-06T18:27:58.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:58.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:58.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:57 smithi107 ceph-mon[32381]: pgmap v2008: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:58.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:57 smithi107 ceph-mon[32381]: osdmap e675: 12 total, 3 up, 11 in 2024-08-06T18:27:58.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:27:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:27:58.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:27:58.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:58.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:58.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:58.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:58.430 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:27:58.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:58.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:58.769 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:58.770 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:58.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:58.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:58 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:58.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:58.798 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:58 smithi117 ceph-mon[32835]: osdmap e676: 12 total, 3 up, 11 in 2024-08-06T18:27:58.848 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:58 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:58.848 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:58.848 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:58 smithi107 ceph-mon[32381]: osdmap e676: 12 total, 3 up, 11 in 2024-08-06T18:27:58.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:27:59.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:58 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:27:59.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1193832697' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:27:59.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:58 smithi086 ceph-mon[29229]: osdmap e676: 12 total, 3 up, 11 in 2024-08-06T18:27:59.036 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:27:59.036 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:27:59.036 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:27:59.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:27:59.054 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:27:59.054 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:27:59.055 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:27:59.055 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:27:59.055 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:27:59.072 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:27:59.073 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:27:59.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:27:59.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:27:59.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:27:59.073 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:27:59.074 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:27:59.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:59.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:27:59.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:27:59.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:27:59.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:59.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:27:59.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:27:59.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:59.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:59.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:27:59.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:27:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:27:59.796 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: pgmap v2011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:59.796 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:27:59.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:27:59.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: osdmap e677: 12 total, 3 up, 11 in 2024-08-06T18:27:59.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/929874426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:59.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:59.797 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:27:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:59.827 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:27:59.831 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:27:59.961 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: pgmap v2011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: osdmap e677: 12 total, 3 up, 11 in 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/929874426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:59.986 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:27:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:27:59.987 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:00.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: pgmap v2011: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:28:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:28:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: osdmap e677: 12 total, 3 up, 11 in 2024-08-06T18:28:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/929874426' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:00.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:27:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:00.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:00.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:00.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:00.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:00.512 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:00.558 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:00.558 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:00.558 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:00.559 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:28:00.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:00.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:00.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:00.778 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:00.814 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:00.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:28:00.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:00.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:00.904 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:01.013 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:01.035 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:01.086 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:01.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:01.168 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:01.168 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:01.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:01.340 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:01.340 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:01.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:01.539 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:01.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:01.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:01.825 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:01 smithi117 ceph-mon[32835]: pgmap v2013: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:01.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:28:01.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:02.065 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:01 smithi086 ceph-mon[29229]: pgmap v2013: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:02.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:02.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:02.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:01 smithi107 ceph-mon[32381]: pgmap v2013: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:02.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:02.417 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:02.459 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:02.460 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:02.465 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:02.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:02.722 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:02.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:02.846 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:02.846 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:02.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:02.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:28:02.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:03.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:03.257 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:03.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:03.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:03.528 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:03.528 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:03.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:03.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:03.788 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:28:03.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:03.874 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:03 smithi117 ceph-mon[32835]: pgmap v2014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:03.875 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:03.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:03 smithi107 ceph-mon[32381]: pgmap v2014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:03.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:04.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:03 smithi086 ceph-mon[29229]: pgmap v2014: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:04.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:04.210 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:04.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:04.474 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:04.609 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:04.610 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:04.626 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:04.627 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:04.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:04.686 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:04.773 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:04.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:05.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:05.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:04 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:05.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:05.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:04 smithi117 ceph-mon[32835]: osdmap e678: 12 total, 3 up, 11 in 2024-08-06T18:28:05.061 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/360534303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:04 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:04 smithi086 ceph-mon[29229]: osdmap e678: 12 total, 3 up, 11 in 2024-08-06T18:28:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/360534303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:05.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:05.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:04 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:05.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1107757919' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:05.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:04 smithi107 ceph-mon[32381]: osdmap e678: 12 total, 3 up, 11 in 2024-08-06T18:28:05.195 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/360534303' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:05.232 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:05.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:28:05.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:05.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:05.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:05.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:05.483 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:05.565 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:05.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:05.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:05.677 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:05.678 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:05.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:05.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:05 smithi117 ceph-mon[32835]: pgmap v2016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:05.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:05.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:05 smithi107 ceph-mon[32381]: pgmap v2016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:05.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:06.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:05 smithi086 ceph-mon[29229]: pgmap v2016: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:06.231 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:06.256 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:06.256 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:06.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:06.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:06.634 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:06.659 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:06.660 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:06.769 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:06.881 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:06.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:06.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:06.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:07.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:07.173 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:07.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:28:07.302 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:07.311 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:07.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:07.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:07.444 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:07.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:28:07.546 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:07.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:07.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:28:07.690 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:07.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:07.833 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:07.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:07 smithi086 ceph-mon[29229]: pgmap v2017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:07.930 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:07 smithi086 ceph-mon[29229]: osdmap e679: 12 total, 3 up, 11 in 2024-08-06T18:28:07.930 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:07.930 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:07.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:08.012 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:07 smithi117 ceph-mon[32835]: pgmap v2017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:08.012 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:07 smithi117 ceph-mon[32835]: osdmap e679: 12 total, 3 up, 11 in 2024-08-06T18:28:08.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:07 smithi107 ceph-mon[32381]: pgmap v2017: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:08.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:07 smithi107 ceph-mon[32381]: osdmap e679: 12 total, 3 up, 11 in 2024-08-06T18:28:08.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:08.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:08.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:08.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:08.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:08.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:08.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:08.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:08.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:08.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:09.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:09.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:09.203 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:09.294 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:09.397 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:09.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:09.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:09.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:09.778 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:28:09.793 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:09.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:28:09.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:09.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:09 smithi107 ceph-mon[32381]: pgmap v2019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:09.853 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:09.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:10.036 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:09 smithi086 ceph-mon[29229]: pgmap v2019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:10.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:10.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:10.086 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:10.098 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:09 smithi117 ceph-mon[32835]: pgmap v2019: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:10.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:10.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:10.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:10.626 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:10.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:10.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:11.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:11.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:10 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:11.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:11.046 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:10 smithi117 ceph-mon[32835]: osdmap e680: 12 total, 3 up, 11 in 2024-08-06T18:28:11.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:10 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:11.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:11.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:10 smithi086 ceph-mon[29229]: osdmap e680: 12 total, 3 up, 11 in 2024-08-06T18:28:11.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:10 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3473660245' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:10 smithi107 ceph-mon[32381]: osdmap e680: 12 total, 3 up, 11 in 2024-08-06T18:28:11.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:11.307 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:11.307 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:11.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:11.307 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:11.425 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:11.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:11.574 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:11.578 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:11.579 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:11.579 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:11.579 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:11.688 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:11.752 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:11.753 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:11.753 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:11.762 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:11.763 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:11.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:11.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:11.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:11.784 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:11.784 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:11.785 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:11.785 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:11.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:11.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:11.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:11.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:11.842 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:11.843 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:11.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:11 smithi086 ceph-mon[29229]: pgmap v2021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:11.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:11 smithi086 ceph-mon[29229]: from='client.19095 172.21.15.86:0/1501918722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:11.843 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:11 smithi086 ceph-mon[29229]: osdmap e681: 12 total, 3 up, 11 in 2024-08-06T18:28:11.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:12.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:12.102 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:11 smithi117 ceph-mon[32835]: pgmap v2021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:12.102 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:11 smithi117 ceph-mon[32835]: from='client.19095 172.21.15.86:0/1501918722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:12.102 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:11 smithi117 ceph-mon[32835]: osdmap e681: 12 total, 3 up, 11 in 2024-08-06T18:28:12.130 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:12.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:28:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:11 smithi107 ceph-mon[32381]: pgmap v2021: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:11 smithi107 ceph-mon[32381]: from='client.19095 172.21.15.86:0/1501918722' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:11 smithi107 ceph-mon[32381]: osdmap e681: 12 total, 3 up, 11 in 2024-08-06T18:28:12.215 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:12.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:12.417 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:12.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:12.503 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:12.504 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:12.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:28:12.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:12.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:12.800 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:12.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:13.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:13.034 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:13.035 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:12 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:13.035 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.035 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.035 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.058 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:13.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:13.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:13.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:13.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:13.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:13.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:13.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:13.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:13.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:13.805 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:13.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:13.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:13 smithi117 ceph-mon[32835]: pgmap v2023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:13.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:13.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:13 smithi107 ceph-mon[32381]: pgmap v2023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:14.082 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:14.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:13 smithi086 ceph-mon[29229]: pgmap v2023: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:14.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:14.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:14.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:28:14.483 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:14.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:28:14.509 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:14.509 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:14.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:14.684 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:14.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:14.754 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:14.796 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:28:14.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:15.015 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:15.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:15.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:15.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:15.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:15.407 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:15.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:15.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:15.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:15 smithi117 ceph-mon[32835]: pgmap v2024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:15 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:15.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:15 smithi117 ceph-mon[32835]: osdmap e682: 12 total, 3 up, 11 in 2024-08-06T18:28:15.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:16.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:15 smithi086 ceph-mon[29229]: pgmap v2024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:16.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:15 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:16.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:16.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:15 smithi086 ceph-mon[29229]: osdmap e682: 12 total, 3 up, 11 in 2024-08-06T18:28:16.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:16.083 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:16.093 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:15 smithi107 ceph-mon[32381]: pgmap v2024: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:16.093 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:15 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:16.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1024296560' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:16.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:15 smithi107 ceph-mon[32381]: osdmap e682: 12 total, 3 up, 11 in 2024-08-06T18:28:16.203 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:16.333 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:16.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:16.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:16.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:16.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:28:16.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:16.828 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:16.829 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:16.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:16.946 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:17.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:17.063 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:17.064 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:17.064 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:17.080 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:17.081 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:17.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:17.081 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:17.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:17.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:17.082 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:17.098 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:17.099 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:17.099 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:17.099 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:17.099 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:17.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:17.100 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:17.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:17.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:17.405 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:17.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:17.432 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:17.542 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:17.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:17.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:17.747 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:17.748 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:17.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:17.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:17.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:17 smithi117 ceph-mon[32835]: pgmap v2026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:17.956 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:17 smithi117 ceph-mon[32835]: osdmap e683: 12 total, 3 up, 11 in 2024-08-06T18:28:17.957 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2794519975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:17.957 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:17.957 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:17 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.067 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:18.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:28:18.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:17 smithi086 ceph-mon[29229]: pgmap v2026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:18.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:17 smithi086 ceph-mon[29229]: osdmap e683: 12 total, 3 up, 11 in 2024-08-06T18:28:18.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2794519975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:18.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.089 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:17 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:17 smithi107 ceph-mon[32381]: pgmap v2026: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:17 smithi107 ceph-mon[32381]: osdmap e683: 12 total, 3 up, 11 in 2024-08-06T18:28:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2794519975' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:17 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:18.341 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:18.363 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:18.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:18.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:18.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:18.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:18.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:18.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:18.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:18.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:18.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:18.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:18.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:28:18.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:19.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:19.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:19.137 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:19.137 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:19.229 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:19.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:19.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:19.402 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:19.402 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:19.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:19.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:19.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:28:19.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:19.772 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:19.772 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:19.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:19 smithi117 ceph-mon[32835]: pgmap v2028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:19.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:20.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:20.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:19 smithi086 ceph-mon[29229]: pgmap v2028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:20.070 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:20.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:20.185 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:19 smithi107 ceph-mon[32381]: pgmap v2028: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:20.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:20.316 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:20.316 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:20.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:20.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:20.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:20.692 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:20.790 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:28:20.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:20.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:20.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:20.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:20.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:21.056 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:21.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:21.086 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:21.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:21.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:28:21.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:21.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:21.568 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:21.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:21.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:21.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:21.785 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:21.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:21.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:21.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:21.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:21 smithi107 ceph-mon[32381]: pgmap v2029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:21.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:21 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:21.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:21.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:21 smithi107 ceph-mon[32381]: osdmap e684: 12 total, 3 up, 11 in 2024-08-06T18:28:21.905 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:21 smithi107 ceph-mon[32381]: osdmap e685: 12 total, 3 up, 11 in 2024-08-06T18:28:21.906 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:21 smithi117 ceph-mon[32835]: pgmap v2029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:21.906 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:21 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:21.906 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:21.906 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:21 smithi117 ceph-mon[32835]: osdmap e684: 12 total, 3 up, 11 in 2024-08-06T18:28:21.906 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:21 smithi117 ceph-mon[32835]: osdmap e685: 12 total, 3 up, 11 in 2024-08-06T18:28:22.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:22.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:21 smithi086 ceph-mon[29229]: pgmap v2029: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:22.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:21 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:22.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2213310345' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:22.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:21 smithi086 ceph-mon[29229]: osdmap e684: 12 total, 3 up, 11 in 2024-08-06T18:28:22.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:21 smithi086 ceph-mon[29229]: osdmap e685: 12 total, 3 up, 11 in 2024-08-06T18:28:22.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:22.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:22.318 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:22.318 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:22.415 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:22.416 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:22.416 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:22.416 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:22.432 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:22.433 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:22.450 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:22.451 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:22.451 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:22.451 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:22.451 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:22.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:22.452 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:22.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:22.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:22.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:22.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:22.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:22.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:22.949 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/719527234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:23.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:23.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/719527234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:23.148 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:23.183 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:23.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:23.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:23.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/719527234' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:23.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:23.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:23.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:28:23.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:23.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:23.763 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:23.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:28:23.790 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:23.870 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:23 smithi107 ceph-mon[32381]: pgmap v2032: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:23.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:24.043 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:24.044 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:24.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:23 smithi086 ceph-mon[29229]: pgmap v2032: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:24.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.053 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:23 smithi117 ceph-mon[32835]: pgmap v2032: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:24.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:24.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:24.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:24.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:24.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:24.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:24.938 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:25.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:25.090 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:25.090 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:25.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:28:25.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:25.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:25.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:25.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:25.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:25.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:25.506 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:25.550 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:25.601 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:25.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:25.665 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:25.665 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:25.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:25.800 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:28:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:25 smithi117 ceph-mon[32835]: pgmap v2033: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:25.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:25.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:25.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:25 smithi086 ceph-mon[29229]: pgmap v2033: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:25.929 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:26.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:26.155 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:26.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:28:26.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:25 smithi107 ceph-mon[32381]: pgmap v2033: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:26.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:26.405 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:28:26.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:26.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:26.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:26.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:26.890 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:26 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:26.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:26.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:26 smithi086 ceph-mon[29229]: osdmap e686: 12 total, 3 up, 11 in 2024-08-06T18:28:26.891 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:26 smithi086 ceph-mon[29229]: osdmap e687: 12 total, 3 up, 11 in 2024-08-06T18:28:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:26 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:26.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:26 smithi107 ceph-mon[32381]: osdmap e686: 12 total, 3 up, 11 in 2024-08-06T18:28:26.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:26 smithi107 ceph-mon[32381]: osdmap e687: 12 total, 3 up, 11 in 2024-08-06T18:28:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:26.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:26.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:26 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1548141649' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:26 smithi117 ceph-mon[32835]: osdmap e686: 12 total, 3 up, 11 in 2024-08-06T18:28:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:26 smithi117 ceph-mon[32835]: osdmap e687: 12 total, 3 up, 11 in 2024-08-06T18:28:27.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:27.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:27.281 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:27.282 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:27.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:27.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:27.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:27.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:27.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:27.724 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:27.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:27.877 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:27 smithi086 ceph-mon[29229]: pgmap v2035: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:27.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:27.981 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:27.982 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:27.982 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:27.986 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:27.987 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:27.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:27.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:27.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:28.004 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:28.005 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:28.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:28.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:27 smithi107 ceph-mon[32381]: pgmap v2035: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:27 smithi117 ceph-mon[32835]: pgmap v2035: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:28.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:28.329 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:28.329 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:28:28.330 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:28.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:28.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:28.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:28:28.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:28.731 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:28.732 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:28.823 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:28.860 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:28.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/774916381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:28.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:28.886 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:28.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/774916381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:28.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:28.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:28.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/774916381' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:29.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:29.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:29.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:29.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:29.550 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:29.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:29.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:29.926 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:29.934 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:29.934 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:29.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:29.935 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:29 smithi086 ceph-mon[29229]: pgmap v2037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:30.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:30.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:30.079 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:29 smithi117 ceph-mon[32835]: pgmap v2037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:30.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:29 smithi107 ceph-mon[32381]: pgmap v2037: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:30.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:30.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:30.462 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:30.462 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:30.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:30.682 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:30.712 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:30.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:28:30.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:30.755 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:30.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:30.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:31.005 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:31.005 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:31.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:31.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:31.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:31.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:31.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:31.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:31.898 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:31.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:28:31.921 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:32.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:31 smithi086 ceph-mon[29229]: pgmap v2038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:32.032 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:32.032 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:32.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:31 smithi117 ceph-mon[32835]: pgmap v2038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:32.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:31 smithi107 ceph-mon[32381]: pgmap v2038: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:32.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:32.323 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:32.324 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:32.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:32.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:32.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:32.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:32.578 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:32.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:32.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:32.981 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:33.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:33.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:33.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:33.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:33.524 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:33.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:33.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:33.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:33.820 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:28:33.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:33.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:34.114 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:33 smithi086 ceph-mon[29229]: pgmap v2039: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:33 smithi117 ceph-mon[32835]: pgmap v2039: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:34.139 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:33 smithi107 ceph-mon[32381]: pgmap v2039: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:34.139 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:34.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:34.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:34.484 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:34.484 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:34.518 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:34.652 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:34.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:34.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:35.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:35.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:34 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:35.077 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:35.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:34 smithi117 ceph-mon[32835]: osdmap e688: 12 total, 3 up, 11 in 2024-08-06T18:28:35.078 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2682255748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:35.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:34 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:35.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:35.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:34 smithi086 ceph-mon[29229]: osdmap e688: 12 total, 3 up, 11 in 2024-08-06T18:28:35.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2682255748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:35.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:35.211 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:34 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:28:35.211 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2333128276' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:34 smithi107 ceph-mon[32381]: osdmap e688: 12 total, 3 up, 11 in 2024-08-06T18:28:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2682255748' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:35.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:35.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:35.586 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:35.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:35.813 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:35.813 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:35.813 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:35.814 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:35.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:35.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:35.815 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:35.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:35 smithi117 ceph-mon[32835]: pgmap v2041: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:35.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:35.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:36.014 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:35 smithi107 ceph-mon[32381]: pgmap v2041: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:36.014 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.014 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.014 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:35 smithi086 ceph-mon[29229]: pgmap v2041: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:36.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:36.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:36.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:36.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:36.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:36.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:36.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:36.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:36.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:36.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:36.950 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:36.950 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:36.980 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:36.980 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:37.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:37.095 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:37.095 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:37.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:37.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:37.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:28:37.407 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:37.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:37.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:37.623 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:37.623 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:37.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:37.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:37.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:37.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:37.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:38.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:37 smithi107 ceph-mon[32381]: pgmap v2042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:38.024 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:37 smithi107 ceph-mon[32381]: osdmap e689: 12 total, 3 up, 11 in 2024-08-06T18:28:38.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:37 smithi086 ceph-mon[29229]: pgmap v2042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:38.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:37 smithi086 ceph-mon[29229]: osdmap e689: 12 total, 3 up, 11 in 2024-08-06T18:28:38.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:38.111 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:37 smithi117 ceph-mon[32835]: pgmap v2042: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:37 smithi117 ceph-mon[32835]: osdmap e689: 12 total, 3 up, 11 in 2024-08-06T18:28:38.134 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:38.138 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:38.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:28:38.224 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:38.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:38.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:38.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:38.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:38.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:38.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:38.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:28:39.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:39.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:39.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:39.464 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:39.504 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:39.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:39.614 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:39.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:39.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:39.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:39 smithi086 ceph-mon[29229]: pgmap v2044: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:39.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:40.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:40.032 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:39 smithi107 ceph-mon[32381]: pgmap v2044: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:39 smithi117 ceph-mon[32835]: pgmap v2044: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:40.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:40.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:40.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:40.542 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:40.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:40.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:40.807 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:28:40.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:40.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:40.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:41.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:41.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:41.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:28:41.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:41.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:41.348 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:41.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:41.488 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:41.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:41.645 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:41.645 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:41.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:41.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:41.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:41 smithi107 ceph-mon[32381]: pgmap v2045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:41.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:41.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:41.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:41 smithi107 ceph-mon[32381]: osdmap e690: 12 total, 3 up, 11 in 2024-08-06T18:28:41.913 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:41 smithi107 ceph-mon[32381]: osdmap e691: 12 total, 3 up, 11 in 2024-08-06T18:28:42.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:42.044 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:42.044 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:42.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:41 smithi086 ceph-mon[29229]: pgmap v2045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:42.044 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:42.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:42.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:41 smithi086 ceph-mon[29229]: osdmap e690: 12 total, 3 up, 11 in 2024-08-06T18:28:42.045 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:41 smithi086 ceph-mon[29229]: osdmap e691: 12 total, 3 up, 11 in 2024-08-06T18:28:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:41 smithi117 ceph-mon[32835]: pgmap v2045: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/534672394' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:28:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:41 smithi117 ceph-mon[32835]: osdmap e690: 12 total, 3 up, 11 in 2024-08-06T18:28:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:41 smithi117 ceph-mon[32835]: osdmap e691: 12 total, 3 up, 11 in 2024-08-06T18:28:42.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:42.289 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:42.295 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:42.295 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:42.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:42.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:42.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:42.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:42.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:42.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:43.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:43.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:43.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:43.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:43.251 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:28:43.357 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:43.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:43.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:43.384 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:43.384 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:43.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:43.628 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:43.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:43.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:43.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:43 smithi086 ceph-mon[29229]: pgmap v2048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:43.895 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:43.935 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:28:43.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:28:44.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:44.060 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:43 smithi107 ceph-mon[32381]: pgmap v2048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:44.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:28:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:43 smithi117 ceph-mon[32835]: pgmap v2048: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:44.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:44.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:44.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:44.459 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:44.459 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:44.476 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:44.476 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:44.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:44.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:44.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:44.498 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:44.499 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:44.499 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:44.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:44.499 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:44.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:44.500 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:44.501 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:44.566 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:44.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:44.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:44 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:28:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:44 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:28:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:44 smithi107 ceph-mon[32381]: osdmap e692: 12 total, 3 up, 11 in 2024-08-06T18:28:44.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/790332871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:44.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:44.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:44.971 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:45.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:44 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:28:45.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:44 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:28:45.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:44 smithi086 ceph-mon[29229]: osdmap e692: 12 total, 3 up, 11 in 2024-08-06T18:28:45.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/790332871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:45.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:44 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:28:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:44 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:28:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:44 smithi117 ceph-mon[32835]: osdmap e692: 12 total, 3 up, 11 in 2024-08-06T18:28:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/790332871' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:45.166 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:45.167 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:45.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:45.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:45.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:45.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:45.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:45.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:45 smithi107 ceph-mon[32381]: pgmap v2050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:45 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:46.066 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:46.066 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:46.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:45 smithi086 ceph-mon[29229]: pgmap v2050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:46.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:45 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:45 smithi117 ceph-mon[32835]: pgmap v2050: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 331 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:45 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:46.177 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:46.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:46.330 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:46.330 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:46.374 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:46.440 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:46.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:46.594 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:46.594 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:46.594 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:46.594 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:46.720 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:46.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:46.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:47.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:47.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:28:47.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:47.392 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:47.452 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:28:47.452 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:28:47.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:47.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:47.795 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:47.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:47 smithi107 ceph-mon[32381]: pgmap v2051: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:47.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:47 smithi107 ceph-mon[32381]: osdmap e693: 12 total, 3 up, 11 in 2024-08-06T18:28:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:47.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:48.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:48.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:47 smithi086 ceph-mon[29229]: pgmap v2051: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:48.076 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:47 smithi086 ceph-mon[29229]: osdmap e693: 12 total, 3 up, 11 in 2024-08-06T18:28:48.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:47 smithi117 ceph-mon[32835]: pgmap v2051: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:48.088 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:47 smithi117 ceph-mon[32835]: osdmap e693: 12 total, 3 up, 11 in 2024-08-06T18:28:48.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:48.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:48.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:48.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:48.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:28:48.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:48.868 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:49.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:49.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:49.211 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:49.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:49.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:49.458 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:49.458 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:49.540 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:49.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:49.786 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:28:49.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:49.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:49 smithi107 ceph-mon[32381]: pgmap v2053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:49.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:49.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:49.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:49 smithi086 ceph-mon[29229]: pgmap v2053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:49.964 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:50.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:49 smithi117 ceph-mon[32835]: pgmap v2053: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:28:50.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:50.342 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:50.359 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:50.359 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:50.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:50.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:50.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:50.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:50.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:51.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:50 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:51.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:51.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:50 smithi086 ceph-mon[29229]: osdmap e694: 12 total, 3 up, 11 in 2024-08-06T18:28:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:50 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:50 smithi117 ceph-mon[32835]: osdmap e694: 12 total, 3 up, 11 in 2024-08-06T18:28:51.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:50 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2844377470' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:28:51.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:50 smithi107 ceph-mon[32381]: osdmap e694: 12 total, 3 up, 11 in 2024-08-06T18:28:51.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:51.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:51.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:51.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:51.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:51.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:51 smithi107 ceph-mon[32381]: pgmap v2055: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:51 smithi107 ceph-mon[32381]: osdmap e695: 12 total, 3 up, 11 in 2024-08-06T18:28:51.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:51 smithi086 ceph-mon[29229]: pgmap v2055: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:51.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:51 smithi086 ceph-mon[29229]: osdmap e695: 12 total, 3 up, 11 in 2024-08-06T18:28:51.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:51.983 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:52.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:51 smithi117 ceph-mon[32835]: pgmap v2055: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:51 smithi117 ceph-mon[32835]: osdmap e695: 12 total, 3 up, 11 in 2024-08-06T18:28:52.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:52.310 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:52.311 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:52.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:52.327 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:52.328 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:52.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:52.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:52.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:52.345 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:52.345 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:52.345 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:52.346 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:52.346 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:52.346 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:52.346 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:52.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:52.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:52.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:52.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:52.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:52.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:52.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/508362291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:52.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:52.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:52.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:52.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:53.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/508362291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:53.144 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:53.144 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/508362291' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:28:53.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:53.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:53.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:53.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:28:53.485 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:53.485 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:53.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:53.562 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:28:53.563 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:28:53.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:53.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:53.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:54.015 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:28:54.016 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:28:54.016 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:28:54.016 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:28:54.016 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:28:54.025 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:54.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:53 smithi086 ceph-mon[29229]: pgmap v2057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:54.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.050 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:54.082 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:54.082 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:54.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:53 smithi107 ceph-mon[32381]: pgmap v2057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:54.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.082 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:53 smithi117 ceph-mon[32835]: pgmap v2057: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:28:54.191 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:54.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:54.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:54.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:54.671 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:28:54.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:54.866 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:54.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:55.133 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:55.153 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:55.153 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:55.269 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:55.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:55.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:55.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:28:55.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:55.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:55.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:55.862 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:55.934 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:55.955 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:55 smithi117 ceph-mon[32835]: pgmap v2058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:55.955 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:55 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:28:56.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:56.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:55 smithi086 ceph-mon[29229]: pgmap v2058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:56.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:55 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:28:56.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:56.209 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:55 smithi107 ceph-mon[32381]: pgmap v2058: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:56.209 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:55 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:28:56.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:56.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:56.459 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:56.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:56.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:56.809 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:28:56.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:28:56.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:57.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:57.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:57.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:57.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:57.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:57.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:28:57.893 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:28:57.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:28:58.035 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:58.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:28:58.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:28:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: pgmap v2059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:28:58.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:58.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:58.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: pgmap v2059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:28:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.299 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: pgmap v2059: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:28:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:28:58.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:58.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:58.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:58.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:28:58.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:58.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:28:58.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:28:58.785 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:28:58.785 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:28:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:28:58.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:28:58.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:59.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:58 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:59.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:59.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:58 smithi086 ceph-mon[29229]: osdmap e696: 12 total, 3 up, 11 in 2024-08-06T18:28:59.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:28:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:58 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:58 smithi117 ceph-mon[32835]: osdmap e696: 12 total, 3 up, 11 in 2024-08-06T18:28:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:58 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:28:59.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1780377760' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:28:59.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:58 smithi107 ceph-mon[32381]: osdmap e696: 12 total, 3 up, 11 in 2024-08-06T18:28:59.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:28:59.298 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:553,from:2,reservation_nonce:22)", 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 516.83030857200004, 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012249200000000001, 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.299 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387608+0000", 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.300 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387609+0000", 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.48e-06 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387617+0000", 2024-08-06T18:28:59.301 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9510000000000006e-06 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387631+0000", 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4141000000000001e-05 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.302 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387640+0000", 2024-08-06T18:28:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.3799999999999994e-06 2024-08-06T18:28:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.303 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387689+0000", 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9407000000000003e-05 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387705+0000", 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5583e-05 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.304 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387730+0000", 2024-08-06T18:28:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5550000000000001e-05 2024-08-06T18:28:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.305 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:553/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 516.82974902000001, 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012141, 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.306 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388167+0000", 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.307 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388168+0000", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.9500000000000002e-07 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388172+0000", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2220000000000004e-06 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388181+0000", 2024-08-06T18:28:59.308 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0610000000000002e-06 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388188+0000", 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.153e-06 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388247+0000", 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8544000000000001e-05 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.309 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388258+0000", 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0879999999999999e-05 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388289+0000", 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0954999999999998e-05 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.310 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:553,from:2,reservation_nonce:0)", 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 516.82907031800005, 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110738, 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.311 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388846+0000", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4999999999999999e-07 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388848+0000", 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7349999999999999e-06 2024-08-06T18:28:59.312 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388854+0000", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5450000000000003e-06 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388859+0000", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0520000000000001e-06 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388937+0000", 2024-08-06T18:28:59.313 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8552000000000003e-05 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388949+0000", 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1409e-05 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.314 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388957+0000", 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1950000000000003e-06 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:553,from:6,reservation_nonce:24)", 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 513.81367585099997, 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014068199999999999, 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.315 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404240+0000", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.316 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404242+0000", 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5540000000000001e-06 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404251+0000", 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.2010000000000005e-06 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.317 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404270+0000", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9028000000000001e-05 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404279+0000", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5220000000000007e-06 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404340+0000", 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1649000000000006e-05 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.318 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404355+0000", 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4484999999999999e-05 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404381+0000", 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6242999999999999e-05 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.319 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:553/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 513.81293304300004, 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000101147, 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.320 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404983+0000", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404984+0000", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.58e-07 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.404995+0000", 2024-08-06T18:28:59.321 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.095e-05 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405004+0000", 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.3200000000000006e-06 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405009+0000", 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0930000000000002e-06 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.322 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405053+0000", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3495000000000003e-05 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405060+0000", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4619999999999999e-06 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:25.405084+0000", 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4369e-05 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.323 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:575,from:6,reservation_nonce:25)", 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 431.64028503100002, 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020588599999999999, 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.324 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.325 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577634+0000", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2400000000000002e-06 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577646+0000", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2357e-05 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577680+0000", 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4119000000000003e-05 2024-08-06T18:28:59.326 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577696+0000", 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5716e-05 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577796+0000", 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010001900000000001 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.327 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577811+0000", 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5299e-05 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577837+0000", 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6135999999999998e-05 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.328 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:575/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 431.63913653100002, 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028445800000000002, 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.329 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578786+0000", 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5400000000000001e-06 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578798+0000", 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1236000000000001e-05 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578827+0000", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9070000000000001e-05 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578842+0000", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4916000000000001e-05 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578978+0000", 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136534 2024-08-06T18:28:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578998+0000", 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9412999999999999e-05 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.579064+0000", 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6748999999999994e-05 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:575,from:6,reservation_nonce:0)", 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 431.63781936399999, 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016099300000000001, 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580098+0000", 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3300000000000002e-07 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580101+0000", 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6940000000000001e-06 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580130+0000", 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8833e-05 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580136+0000", 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7520000000000002e-06 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580238+0000", 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102543 2024-08-06T18:28:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580249+0000", 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0823e-05 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580258+0000", 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8149999999999995e-06 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:599,from:6,reservation_nonce:26)", 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 331.435892427, 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018547099999999999, 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782025+0000", 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5340000000000001e-06 2024-08-06T18:28:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782035+0000", 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999993e-06 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782055+0000", 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0650999999999999e-05 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782064+0000", 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9209999999999998e-06 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782156+0000", 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1822000000000003e-05 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782172+0000", 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5611999999999999e-05 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782209+0000", 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7830999999999997e-05 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:599/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 331.435251075, 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111396, 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782666+0000", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6899999999999998e-07 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782670+0000", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.048e-06 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782697+0000", 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6948999999999999e-05 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782702+0000", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8199999999999996e-06 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782739+0000", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7166999999999998e-05 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782745+0000", 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1430000000000003e-06 2024-08-06T18:28:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782777+0000", 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.18e-05 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:599,from:6,reservation_nonce:0)", 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 331.434558041, 2024-08-06T18:28:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136124, 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783359+0000", 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1699999999999999e-07 2024-08-06T18:28:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783361+0000", 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5359999999999999e-06 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783367+0000", 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.285e-06 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783376+0000", 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4780000000000008e-06 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783463+0000", 2024-08-06T18:28:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6978999999999995e-05 2024-08-06T18:28:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783480+0000", 2024-08-06T18:28:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6634e-05 2024-08-06T18:28:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783494+0000", 2024-08-06T18:28:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4895000000000001e-05 2024-08-06T18:28:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:649,from:6,reservation_nonce:28)", 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 163.11914961900001, 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147861, 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:28:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098768+0000", 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4e-07 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098773+0000", 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9139999999999999e-06 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098783+0000", 2024-08-06T18:28:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0508e-05 2024-08-06T18:28:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098790+0000", 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5409999999999997e-06 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098871+0000", 2024-08-06T18:28:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1824999999999997e-05 2024-08-06T18:28:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098887+0000", 2024-08-06T18:28:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5617e-05 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098915+0000", 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7515999999999999e-05 2024-08-06T18:28:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:649/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 163.11840195799999, 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013723899999999999, 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:28:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099515+0000", 2024-08-06T18:28:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6300000000000005e-07 2024-08-06T18:28:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099522+0000", 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0110000000000003e-06 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099533+0000", 2024-08-06T18:28:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1272e-05 2024-08-06T18:28:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099540+0000", 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4670000000000001e-06 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099619+0000", 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9609000000000003e-05 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099628+0000", 2024-08-06T18:28:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5110000000000003e-06 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099652+0000", 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3706e-05 2024-08-06T18:28:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:649,from:6,reservation_nonce:0)", 2024-08-06T18:28:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.100323+0000", 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 163.11759356900001, 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000100877, 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100323+0000", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100323+0000", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100323+0000", 2024-08-06T18:28:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0899999999999997e-07 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100325+0000", 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8920000000000001e-06 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100329+0000", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3560000000000003e-06 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100333+0000", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8530000000000002e-06 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100404+0000", 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0908000000000006e-05 2024-08-06T18:28:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100415+0000", 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0722e-05 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.100424+0000", 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8370000000000003e-06 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:673,from:2,reservation_nonce:27)", 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 69.787553896999995, 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145129, 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430364+0000", 2024-08-06T18:28:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.513e-06 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430371+0000", 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0190000000000001e-06 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430385+0000", 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430396+0000", 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0229e-05 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430466+0000", 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0858000000000005e-05 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430481+0000", 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4596999999999999e-05 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430508+0000", 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:673/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 69.786946154000006, 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111407, 2024-08-06T18:28:59.385 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:28:59.386 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430971+0000", 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6599999999999996e-07 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430975+0000", 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2889999999999999e-06 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.387 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430986+0000", 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0729e-05 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430992+0000", 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.694e-06 2024-08-06T18:28:59.388 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431040+0000", 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8418999999999998e-05 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431050+0000", 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.389 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431082+0000", 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1754999999999997e-05 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:673,from:2,reservation_nonce:0)", 2024-08-06T18:28:59.390 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 69.786332926, 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012740599999999999, 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.391 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:28:59.392 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431584+0000", 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.36e-07 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:28:59.393 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431586+0000", 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0080000000000001e-06 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431590+0000", 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0090000000000001e-06 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.394 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431596+0000", 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2759999999999997e-06 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431681+0000", 2024-08-06T18:28:59.395 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4974000000000003e-05 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431696+0000", 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4654000000000001e-05 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:28:59.396 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431711+0000", 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5248999999999999e-05 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.397 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:28:59.398 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:28:59.398 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:28:59.398 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:28:59.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:28:59.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:28:59.734 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:28:59.734 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:28:59.734 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:28:59.734 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:28:59.734 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:28:59.735 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:28:59.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:28:59.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:00.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:59 smithi086 ceph-mon[29229]: pgmap v2061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:00.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3479720013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:00.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:28:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:00.090 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:00.090 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:28:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:00.090 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:59 smithi107 ceph-mon[32381]: pgmap v2061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:00.091 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3479720013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:00.091 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.091 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:28:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:00.225 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:59 smithi117 ceph-mon[32835]: pgmap v2061: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:00.225 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3479720013' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:00.226 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.226 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:28:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:00.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:00.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:00.478 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:00.478 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:00.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:00.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:00.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:00.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:00.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:00.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:29:01.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:01.029 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:01.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:01.203 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:01.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:01.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:01.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:01.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:01.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:01.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:01.669 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:01.727 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:01.805 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:01.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:01 smithi107 ceph-mon[32381]: pgmap v2062: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:01 smithi107 ceph-mon[32381]: osdmap e697: 12 total, 3 up, 11 in 2024-08-06T18:29:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:01.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:02.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:02.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:01 smithi086 ceph-mon[29229]: pgmap v2062: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:02.107 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:01 smithi086 ceph-mon[29229]: osdmap e697: 12 total, 3 up, 11 in 2024-08-06T18:29:02.203 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:02.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:02.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:02.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:02.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:02.229 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:01 smithi117 ceph-mon[32835]: pgmap v2062: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:02.229 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:01 smithi117 ceph-mon[32835]: osdmap e697: 12 total, 3 up, 11 in 2024-08-06T18:29:02.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:02.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:02.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:29:02.607 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:02.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:02.874 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:02.966 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:03.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:03.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:03.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:03.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:03.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:03.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:03.437 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:03.437 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:03.548 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:03.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:29:03.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:03.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:03.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:04.085 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:04.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:03 smithi117 ceph-mon[32835]: pgmap v2064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:03 smithi107 ceph-mon[32381]: pgmap v2064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:04.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:04.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:04.405 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:03 smithi086 ceph-mon[29229]: pgmap v2064: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:04.405 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:04.405 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:04.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:04.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:04.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:04.921 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:04.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:04.938 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:04.939 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:04.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:04.956 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:04.956 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:04.957 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:04.957 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:04.958 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:05.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:05.175 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:05.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:05.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:05.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:05.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:05.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:05.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:05.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:05.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:05.873 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:29:05.893 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:05.906 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:05.906 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:06.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:06.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: pgmap v2065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:06.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2965196047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:06.053 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:29:06.054 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:05 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:29:06.154 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:06.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:06.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: pgmap v2065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:06.189 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2965196047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:06.190 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.190 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.190 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.190 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:29:06.190 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:05 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:29:06.277 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: pgmap v2065: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:06.277 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2965196047' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:06.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:06.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:29:06.278 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:05 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:29:06.299 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:06.429 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:06.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:06.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:06.835 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:06.970 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:06.995 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:06.995 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:07.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:07.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:07.364 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:07.364 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:07.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:07.444 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:07.445 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:07.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:07.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:07.765 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:07.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:07.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:08.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:07 smithi086 ceph-mon[29229]: pgmap v2066: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:08.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:08.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:08.200 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:07 smithi117 ceph-mon[32835]: pgmap v2066: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:07 smithi107 ceph-mon[32381]: pgmap v2066: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:08.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:08.442 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:08.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:08.710 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:08.845 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:08.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:08.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:08.852 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:08.853 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:08.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:08.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:08.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:09.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:09.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:09.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:29:09.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:09.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:09.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:09.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:09.916 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:09.916 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:09.923 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: pgmap v2067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: osdmap e698: 12 total, 3 up, 11 in 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=56,logm=2} crc {auth=2560447484,config=507150784,health=639110086,kv=725554375,logm=210317710}) 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403664293}) 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653779613}) 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40746670}) 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669372444}) 2024-08-06T18:29:09.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303038136}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:29:09.938 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604397127}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329736938}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013916591}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385459398}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262787760}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553004107}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190826763}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607867046}) 2024-08-06T18:29:09.939 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2813000377}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233832315}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028893769}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320090718}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671409276}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988050097}) 2024-08-06T18:29:09.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3937295958}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520731132}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727367797}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523006}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156977421}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=623464223}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333018434}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325065095}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628844944}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500713736}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993185054}) 2024-08-06T18:29:09.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430534137}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868835549}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945831984}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646085170}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165147629}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285585842}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071927776}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3838696112}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699428608}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042194197}) 2024-08-06T18:29:09.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917701007}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054172610}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039720384}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3421074062}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2895001528}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983817758}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614343022}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129647466}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913490100}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976808690}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332917369}) 2024-08-06T18:29:09.943 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880012538}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2534582978}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601361044}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315511580}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195761892}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923812248}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=21} crc {logm=2191820213,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4221293093,mon_config_key=2516177309}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1873271235}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2533210233}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1583609488,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1641562001,osdmap=3498553621}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1789421628}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1494541196}) 2024-08-06T18:29:09.944 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3513069607}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2992250724}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2449896557}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1983044524}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=326190608}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=345468727}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=213966696}) 2024-08-06T18:29:09.945 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:09.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:10.051 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:10.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: pgmap v2067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:29:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:29:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: osdmap e698: 12 total, 3 up, 11 in 2024-08-06T18:29:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=56,logm=2} crc {auth=2560447484,config=507150784,health=639110086,kv=725554375,logm=210317710}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403664293}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653779613}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40746670}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669372444}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303038136}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:29:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604397127}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329736938}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013916591}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385459398}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262787760}) 2024-08-06T18:29:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553004107}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190826763}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607867046}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2813000377}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233832315}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028893769}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320090718}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671409276}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988050097}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3937295958}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520731132}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727367797}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523006}) 2024-08-06T18:29:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156977421}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=623464223}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333018434}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325065095}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628844944}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500713736}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993185054}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430534137}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868835549}) 2024-08-06T18:29:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945831984}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646085170}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165147629}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285585842}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071927776}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3838696112}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699428608}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042194197}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917701007}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054172610}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039720384}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3421074062}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2895001528}) 2024-08-06T18:29:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983817758}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614343022}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129647466}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913490100}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976808690}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332917369}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880012538}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2534582978}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601361044}) 2024-08-06T18:29:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315511580}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195761892}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923812248}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=21} crc {logm=2191820213,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4221293093,mon_config_key=2516177309}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1873271235}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2533210233}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1583609488,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1641562001,osdmap=3498553621}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1789421628}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1494541196}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3513069607}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2992250724}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2449896557}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1983044524}) 2024-08-06T18:29:10.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=326190608}) 2024-08-06T18:29:10.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:29:10.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=345468727}) 2024-08-06T18:29:10.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=213966696}) 2024-08-06T18:29:10.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: pgmap v2067: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: osdmap e698: 12 total, 3 up, 11 in 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=28,config=2,health=12,kv=56,logm=2} crc {auth=2560447484,config=507150784,health=639110086,kv=725554375,logm=210317710}) 2024-08-06T18:29:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1403664293}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=653779613}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=40746670}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3669372444}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3303038136}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1228298893}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3030256623}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=990290232}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3310325580}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1653614137}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=783895110}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1924616904}) 2024-08-06T18:29:10.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4053171823}) 2024-08-06T18:29:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2887631668}) 2024-08-06T18:29:10.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=393163884}) 2024-08-06T18:29:10.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2310433126}) 2024-08-06T18:29:10.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=646858509}) 2024-08-06T18:29:10.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=210660103}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3604397127}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2329736938}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2013916591}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2385459398}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2262787760}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1553004107}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=190826763}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2607867046}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2813000377}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2233832315}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2028893769}) 2024-08-06T18:29:10.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3320090718}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1671409276}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3988050097}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3937295958}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2520731132}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3727367797}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3339523006}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4156977421}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=623464223}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1333018434}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3325065095}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2628844944}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1500713736}) 2024-08-06T18:29:10.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1993185054}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=430534137}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1868835549}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3945831984}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3646085170}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4165147629}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1285585842}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1071927776}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3838696112}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=699428608}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2042194197}) 2024-08-06T18:29:10.220 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2917701007}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4054172610}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2039720384}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3421074062}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2895001528}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=983817758}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3614343022}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=129647466}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1913490100}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3976808690}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3332917369}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3880012538}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2534582978}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2601361044}) 2024-08-06T18:29:10.221 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2315511580}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3195761892}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=923812248}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=38,mdsmap=3,mgr=18,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=21} crc {logm=2191820213,mdsmap=817224545,mgr=2408834153,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=4221293093,mon_config_key=2516177309}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1873271235}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=2533210233}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=58,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=24} crc {mon_config_key=1583609488,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1641562001,osdmap=3498553621}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1789421628}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1494541196}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3513069607}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2992250724}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2449896557}) 2024-08-06T18:29:10.222 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1983044524}) 2024-08-06T18:29:10.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=326190608}) 2024-08-06T18:29:10.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:29:10.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=345468727}) 2024-08-06T18:29:10.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=89} crc {osdmap=213966696}) 2024-08-06T18:29:10.223 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:10.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:10.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:10.362 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:10.362 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:10.495 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:10.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:10.688 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:10.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:29:10.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:10.850 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:10.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:10.916 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:10.916 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:11.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:11.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:11.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:10 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:11.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:11.201 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:10 smithi107 ceph-mon[32381]: osdmap e699: 12 total, 3 up, 11 in 2024-08-06T18:29:11.310 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:11.316 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:10 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:11.316 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:11.316 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:10 smithi086 ceph-mon[29229]: osdmap e699: 12 total, 3 up, 11 in 2024-08-06T18:29:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:10 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3211683216' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:11.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:10 smithi117 ceph-mon[32835]: osdmap e699: 12 total, 3 up, 11 in 2024-08-06T18:29:11.416 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:11.445 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:11.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:11.720 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:11.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:11.924 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:11.925 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:11.925 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:11.925 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:11.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:12.117 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:11 smithi107 ceph-mon[32381]: pgmap v2070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:11 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:29:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:11 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:29:12.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:12.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:12.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:11 smithi086 ceph-mon[29229]: pgmap v2070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:11 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:29:12.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:11 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:29:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:11 smithi117 ceph-mon[32835]: pgmap v2070: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:11 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:29:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:11 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:29:12.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:12.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:12.651 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:12.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:12.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:13.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:13.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:13.324 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:13.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:29:13.458 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:13.588 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:13.718 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:13.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:13.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:14.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:14.146 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:13 smithi117 ceph-mon[32835]: pgmap v2071: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:13 smithi107 ceph-mon[32381]: pgmap v2071: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:14.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:14.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:14.260 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:13 smithi086 ceph-mon[29229]: pgmap v2071: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:14.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:14.460 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:14.524 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:14.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:14.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:14.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:14.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:14.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:14.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:14.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:15.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:15.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:15.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:29:15.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:14 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:29:15.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:14 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:29:15.190 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:15.190 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:15.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:15.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:14 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:29:15.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:14 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:29:15.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:15.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:14 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:29:15.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:14 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:29:15.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:15.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:15.428 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:15.430 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:15.452 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:15.453 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:15.453 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:15.453 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:15.454 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:15.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:15.473 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:15.474 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:15.474 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:15.474 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:15.475 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:15.475 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:15.475 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:15.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:15.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:15.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:15.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:16.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:16.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:15 smithi086 ceph-mon[29229]: pgmap v2072: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:16.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:15 smithi086 ceph-mon[29229]: osdmap e700: 12 total, 3 up, 11 in 2024-08-06T18:29:16.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1556838695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:16.039 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:16.142 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:16.168 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:15 smithi107 ceph-mon[32381]: pgmap v2072: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:16.168 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:15 smithi107 ceph-mon[32381]: osdmap e700: 12 total, 3 up, 11 in 2024-08-06T18:29:16.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1556838695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:16.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:16.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:16.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:15 smithi117 ceph-mon[32835]: pgmap v2072: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:16.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:15 smithi117 ceph-mon[32835]: osdmap e700: 12 total, 3 up, 11 in 2024-08-06T18:29:16.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1556838695' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:16.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:16.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:16.425 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:16.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:16.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:16.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:16.876 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:16.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:17.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:17.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:16 smithi117 ceph-mon[32835]: osdmap e701: 12 total, 3 up, 11 in 2024-08-06T18:29:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:16 smithi107 ceph-mon[32381]: osdmap e701: 12 total, 3 up, 11 in 2024-08-06T18:29:17.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:17.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:17.353 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:16 smithi086 ceph-mon[29229]: osdmap e701: 12 total, 3 up, 11 in 2024-08-06T18:29:17.353 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:17.353 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:17.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:17.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:17.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:17.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:17.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:18.032 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:18.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:18.193 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:18.193 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:18.193 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:17 smithi107 ceph-mon[32381]: pgmap v2074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:18.225 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:17 smithi086 ceph-mon[29229]: pgmap v2074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:18.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:18.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:17 smithi117 ceph-mon[32835]: pgmap v2074: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:18.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:29:18.448 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:18.448 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:18.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:18.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:18.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:18.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:18.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:18.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:18.948 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:29:18.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:19.016 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:19.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:19.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:19.288 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:19.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:19.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:19.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:19.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:19.856 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:19.856 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:19.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:20.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:20.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:19 smithi107 ceph-mon[32381]: pgmap v2076: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:20.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:19 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:20.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:20.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:19 smithi107 ceph-mon[32381]: osdmap e702: 12 total, 3 up, 11 in 2024-08-06T18:29:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:19 smithi086 ceph-mon[29229]: pgmap v2076: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:19 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:19 smithi086 ceph-mon[29229]: osdmap e702: 12 total, 3 up, 11 in 2024-08-06T18:29:20.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:20.221 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:29:20.221 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:29:20.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:20.357 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:19 smithi117 ceph-mon[32835]: pgmap v2076: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:20.358 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:19 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:20.358 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1008906281' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:20.358 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:19 smithi117 ceph-mon[32835]: osdmap e702: 12 total, 3 up, 11 in 2024-08-06T18:29:20.419 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:20.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:20.507 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:20.507 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:20.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:20.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:20.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:29:20.895 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:21.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:21.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:21.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:21.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:21.528 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:21.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:21.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:21.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:29:21.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:21.940 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:22.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:22.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:21 smithi086 ceph-mon[29229]: pgmap v2078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:22.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:21 smithi086 ceph-mon[29229]: osdmap e703: 12 total, 3 up, 11 in 2024-08-06T18:29:22.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:21 smithi107 ceph-mon[32381]: pgmap v2078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:21 smithi107 ceph-mon[32381]: osdmap e703: 12 total, 3 up, 11 in 2024-08-06T18:29:22.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:22.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:22.222 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:21 smithi117 ceph-mon[32835]: pgmap v2078: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:22.222 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:21 smithi117 ceph-mon[32835]: osdmap e703: 12 total, 3 up, 11 in 2024-08-06T18:29:22.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:22.466 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:22.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:22.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:22.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:22.818 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:22.868 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:22.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:23.091 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:23.092 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:23.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:23.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:23.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:23.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:23.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:23.531 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:23.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:23.691 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:23.691 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:23.716 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:23.801 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:23.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:23.835 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:23.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:23.937 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:23.938 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:23.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:23.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:23.954 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:23.954 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:23.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:23.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:23.955 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:23.972 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:23.973 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:24.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:24.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:23 smithi086 ceph-mon[29229]: pgmap v2080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:24.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4257569072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:24.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:23 smithi107 ceph-mon[32381]: pgmap v2080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4257569072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:24.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:24.363 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:23 smithi117 ceph-mon[32835]: pgmap v2080: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:24.363 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4257569072' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:24.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:24.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:24.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:24.474 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:29:24.474 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:29:24.474 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:29:24.474 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:29:24.474 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:29:24.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:24.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:24.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:25.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:25.272 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:25.272 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:25.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:25.403 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:25.539 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:25.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:25.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:25.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:25.954 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:29:26.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:25 smithi086 ceph-mon[29229]: pgmap v2081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:26.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:25 smithi107 ceph-mon[32381]: pgmap v2081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:26.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:26.239 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:25 smithi117 ceph-mon[32835]: pgmap v2081: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:26.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:26.386 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:26.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:26.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:26.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:26.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:27.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:27.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:27.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:29:27.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:27.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:27.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:27.658 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:27.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:27.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:27.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:27.991 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:27 smithi117 ceph-mon[32835]: pgmap v2082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:28.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:27 smithi086 ceph-mon[29229]: pgmap v2082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:28.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:27 smithi107 ceph-mon[32381]: pgmap v2082: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:28.232 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:28.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:28.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:28.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:28.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:28.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:29:29.009 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 546.62238482099997, 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017163700000000001, 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387806+0000", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387807+0000", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0500000000000002e-07 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387810+0000", 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3239999999999999e-06 2024-08-06T18:29:29.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387819+0000", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5730000000000008e-06 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387824+0000", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3630000000000002e-06 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387861+0000", 2024-08-06T18:29:29.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6433000000000002e-05 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387875+0000", 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4549000000000001e-05 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.387978+0000", 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010249 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 546.62174543499998, 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.2353000000000003e-05, 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:29:29.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388446+0000", 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4600000000000005e-07 2024-08-06T18:29:29.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388449+0000", 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.174e-06 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388457+0000", 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1240000000000005e-06 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388462+0000", 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4399999999999998e-06 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388493+0000", 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1256e-05 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388500+0000", 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4690000000000002e-06 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:20:22.388518+0000", 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8343999999999999e-05 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:673,from:0,reservation_nonce:0)", 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 99.579617795000004, 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017925499999999999, 2024-08-06T18:29:29.021 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.022 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430574+0000", 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-07 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430577+0000", 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.864e-06 2024-08-06T18:29:29.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430587+0000", 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0581000000000001e-05 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430593+0000", 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7060000000000001e-06 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430636+0000", 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2914000000000003e-05 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430649+0000", 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3098999999999999e-05 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430752+0000", 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330799999999999 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.025 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 99.578969889000007, 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6106999999999996e-05, 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431222+0000", 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4899999999999995e-07 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431225+0000", 2024-08-06T18:29:29.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2949999999999998e-06 2024-08-06T18:29:29.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.028 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431233+0000", 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8660000000000006e-06 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431237+0000", 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1880000000000004e-06 2024-08-06T18:29:29.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431266+0000", 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8912e-05 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431271+0000", 2024-08-06T18:29:29.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5330000000000002e-06 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431287+0000", 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5764e-05 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:697,from:0,reservation_nonce:0)", 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:29:29.031 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.786548606, 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000184933, 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:29:29.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223643+0000", 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0699999999999996e-07 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223646+0000", 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5220000000000002e-06 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223655+0000", 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1509999999999993e-06 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223660+0000", 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3759999999999999e-06 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223696+0000", 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223709+0000", 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3013999999999999e-05 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223827+0000", 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118313 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.035 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 23.785940609000001, 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7805000000000005e-05, 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:29:29.036 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224251+0000", 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8699999999999995e-07 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224254+0000", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7010000000000001e-06 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224261+0000", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224266+0000", 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3579999999999996e-06 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224302+0000", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6529999999999998e-05 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224310+0000", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8839999999999994e-06 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224328+0000", 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7901e-05 2024-08-06T18:29:29.039 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:29:29.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:29:29.040 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:29:29.169 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:29.286 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:29.287 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:29.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:29.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:29.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:29.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:29.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:29.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:29.581 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:29.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:29.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:29.963 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:29:29.982 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:30.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:29 smithi107 ceph-mon[32381]: pgmap v2083: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:30.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:30.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:30.251 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:30.274 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:29 smithi086 ceph-mon[29229]: pgmap v2083: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:30.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:30.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:29:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:29 smithi117 ceph-mon[32835]: pgmap v2083: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:30.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:30.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:30.535 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:30.535 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:30.609 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:30.643 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:30.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:30.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:31.046 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:31.049 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:31.049 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:31.049 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:30 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:31.049 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:31.049 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:30 smithi107 ceph-mon[32381]: osdmap e704: 12 total, 3 up, 11 in 2024-08-06T18:29:31.178 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:31.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:31.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:30 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:31.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:31.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:30 smithi086 ceph-mon[29229]: osdmap e704: 12 total, 3 up, 11 in 2024-08-06T18:29:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:30 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1156704842' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:30 smithi117 ceph-mon[32835]: osdmap e704: 12 total, 3 up, 11 in 2024-08-06T18:29:31.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:31.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:31.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:31.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:31.859 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:31.859 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:31.889 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:31.889 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:31.890 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:31.904 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:31.904 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:31.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:31.904 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:31.905 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:31.921 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:31.922 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:31.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:32.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:32.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:31 smithi107 ceph-mon[32381]: pgmap v2085: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:32.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:31 smithi107 ceph-mon[32381]: osdmap e705: 12 total, 3 up, 11 in 2024-08-06T18:29:32.123 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1121572898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:32.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:31 smithi086 ceph-mon[29229]: pgmap v2085: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:32.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:31 smithi086 ceph-mon[29229]: osdmap e705: 12 total, 3 up, 11 in 2024-08-06T18:29:32.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1121572898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:32.217 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:32.217 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:32.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:31 smithi117 ceph-mon[32835]: pgmap v2085: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:31 smithi117 ceph-mon[32835]: osdmap e705: 12 total, 3 up, 11 in 2024-08-06T18:29:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1121572898' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:32.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:32.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:32.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:32.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:32.519 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:32.520 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:32.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:32.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:32.904 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:32.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:29:32.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:33.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:33.164 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:33.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:33.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:33.247 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:33.247 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:33.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:33.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:33.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:29:33.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:33.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:33.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:33.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:33.657 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:33.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:33.890 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:34.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:34.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:33 smithi086 ceph-mon[29229]: pgmap v2087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:33 smithi117 ceph-mon[32835]: pgmap v2087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:34.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:34.262 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:33 smithi107 ceph-mon[32381]: pgmap v2087: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:34.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:34.425 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:34.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:29:34.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:34.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:34.747 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:34.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:34.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:35.079 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:35.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:35.283 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:35.343 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:35.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:29:35.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:35.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:35.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:35.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:35.592 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:35.635 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:35.635 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:35.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:35.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:35.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:35.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:35.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:35.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:35.961 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:36.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:36.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:36.217 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:36.217 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:36.217 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:35 smithi086 ceph-mon[29229]: pgmap v2088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:36.279 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:35 smithi107 ceph-mon[32381]: pgmap v2088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:36.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:35 smithi117 ceph-mon[32835]: pgmap v2088: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:36.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:36.506 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:36.506 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:36.545 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:36.545 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:36.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:36.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:36.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:37.001 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:29:37.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:37.056 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:37.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:29:37.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:37.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:37.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:37.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:29:37.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:37.294 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:37.294 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:37.295 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:37.296 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:29:37.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:37.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:37.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:37.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:37.705 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:37.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:29:37.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:37.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:37.956 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:37.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:37.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:38.020 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:38.021 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:38.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:38.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:38.234 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: pgmap v2089: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:38.234 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: osdmap e706: 12 total, 3 up, 11 in 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/945615403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:37 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: pgmap v2089: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: osdmap e706: 12 total, 3 up, 11 in 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/945615403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.288 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.289 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:37 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:38.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:38.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:38.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: pgmap v2089: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:38.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:38.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1035728612' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:29:38.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: osdmap e706: 12 total, 3 up, 11 in 2024-08-06T18:29:38.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/945615403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:38.368 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.369 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:37 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:38.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:38.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:38.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:38.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:39.016 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:39.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:39.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:39.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:39.453 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:39.555 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:39.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:39.818 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:39.840 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:39.840 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:39.949 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:40.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:29:40.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:40.091 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:39 smithi107 ceph-mon[32381]: pgmap v2091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:40.115 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:40.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:29:40.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:40.278 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:40.347 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:40.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:39 smithi086 ceph-mon[29229]: pgmap v2091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:40.367 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:40.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:39 smithi117 ceph-mon[32835]: pgmap v2091: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:40.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:40.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:29:40.617 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:40.752 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:40.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:41.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:41.128 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:41.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:41.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:41.286 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:41.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:41.428 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:41.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:41.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:41.687 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:41.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:41.906 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:41.917 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:41.946 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:41.947 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:41.947 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:41.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:41.964 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:41.964 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:41.965 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:41.965 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:41.965 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:41.965 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:41.965 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:41.983 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:41.983 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:41.983 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:41.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:41.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:41.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:41.984 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:42.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:42.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:41 smithi086 ceph-mon[29229]: pgmap v2092: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:42.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:41 smithi086 ceph-mon[29229]: osdmap e707: 12 total, 3 up, 11 in 2024-08-06T18:29:42.216 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1478717023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:42.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:42.245 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:41 smithi107 ceph-mon[32381]: pgmap v2092: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:42.245 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:41 smithi107 ceph-mon[32381]: osdmap e707: 12 total, 3 up, 11 in 2024-08-06T18:29:42.245 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1478717023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:42.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:41 smithi117 ceph-mon[32835]: pgmap v2092: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:41 smithi117 ceph-mon[32835]: osdmap e707: 12 total, 3 up, 11 in 2024-08-06T18:29:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1478717023' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:42.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:42.545 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:42.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:29:42.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:42.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:42.798 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:42.854 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:42.854 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:42.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:43.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:43.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.053 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.053 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.053 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:43.155 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:43.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:43.308 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:43.309 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:43.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:43.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:43.572 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:43.572 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:43.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:43.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:43.739 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:43.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:43.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:44.077 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:43 smithi117 ceph-mon[32835]: pgmap v2094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:44.197 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:43 smithi107 ceph-mon[32381]: pgmap v2094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:44.208 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:44.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:43 smithi086 ceph-mon[29229]: pgmap v2094: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:44.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:44.396 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:44.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:44.660 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:44.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:44.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:29:44.929 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:44.984 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:44.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:29:45.000 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:29:45.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:45.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:45.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:45.234 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:45.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:45.305 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:45.305 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:45.306 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:45.306 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:45.325 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:45.460 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:45.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:45.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:45.846 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:45.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:45.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:46.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:46.249 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:46.250 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:46.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:45 smithi086 ceph-mon[29229]: pgmap v2095: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:46.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:45 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:46.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:46.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:45 smithi086 ceph-mon[29229]: osdmap e708: 12 total, 3 up, 11 in 2024-08-06T18:29:46.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:46.270 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:46 smithi107 ceph-mon[32381]: pgmap v2095: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:46.270 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:46 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:46.270 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:46.270 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:46 smithi107 ceph-mon[32381]: osdmap e708: 12 total, 3 up, 11 in 2024-08-06T18:29:46.270 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:46.271 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:46.285 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:45 smithi117 ceph-mon[32835]: pgmap v2095: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:46.285 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:45 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:29:46.285 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/656674219' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:46.285 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:45 smithi117 ceph-mon[32835]: osdmap e708: 12 total, 3 up, 11 in 2024-08-06T18:29:46.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:46.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:46.658 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:46.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:46.814 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:29:46.814 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:29:46.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:47.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:47.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:47.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:47.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:47.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:47.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:29:47.455 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:47.591 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:47.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:47.658 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:47.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:47.674 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:47.675 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:47.675 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:47.675 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:47.675 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:47.676 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:47.692 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:47.692 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:47.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:47.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:47.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:29:47.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:47.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:48.006 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:48.006 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:48.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:47 smithi086 ceph-mon[29229]: pgmap v2097: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:48.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:47 smithi086 ceph-mon[29229]: osdmap e709: 12 total, 3 up, 11 in 2024-08-06T18:29:48.007 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/495873637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:48.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:47 smithi117 ceph-mon[32835]: pgmap v2097: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:47 smithi117 ceph-mon[32835]: osdmap e709: 12 total, 3 up, 11 in 2024-08-06T18:29:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/495873637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:48.138 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:47 smithi107 ceph-mon[32381]: pgmap v2097: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:48.138 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:47 smithi107 ceph-mon[32381]: osdmap e709: 12 total, 3 up, 11 in 2024-08-06T18:29:48.138 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/495873637' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:48.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:48.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:48.462 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:48.463 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:48.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:48.573 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:48.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:48.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:48.910 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:49.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:49.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:49.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:49.067 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.068 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.128 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.128 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.128 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:49.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:49.225 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:29:49.226 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:29:49.226 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:29:49.226 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:29:49.226 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:29:49.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:49.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:49.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:49.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:49.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:49.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:49.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:49.713 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:49.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:29:49.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:49.965 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:49 smithi086 ceph-mon[29229]: pgmap v2099: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:49.987 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:50.108 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:50.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:49 smithi117 ceph-mon[32835]: pgmap v2099: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:49 smithi107 ceph-mon[32381]: pgmap v2099: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:50.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:50.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:50.250 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:50.386 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:50.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:29:50.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:50.486 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:50.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:50.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:50.711 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:50.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:50.812 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:29:50.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:51.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:51.071 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:51.099 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:51.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:29:51.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:51.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:51.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:51.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:51.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:51.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:51.754 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:51.754 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:51.754 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:51.759 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:553,from:0,reservation_nonce:0)", 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 566.33829445499998, 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000266113, 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.760 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404598+0000", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404599+0000", 2024-08-06T18:29:51.761 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.023e-06 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404602+0000", 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.534e-06 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404616+0000", 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3881e-05 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.762 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404626+0000", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0402e-05 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404697+0000", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0785999999999996e-05 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404716+0000", 2024-08-06T18:29:51.763 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9258000000000001e-05 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.404864+0000", 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000147229 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:29:51.764 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 566.337519875, 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00014078100000000001, 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.765 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405372+0000", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405373+0000", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4000000000000001e-07 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405377+0000", 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.391e-06 2024-08-06T18:29:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405387+0000", 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.0040999999999999e-05 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405393+0000", 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.976e-06 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405481+0000", 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.7995e-05 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405490+0000", 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.1649999999999995e-06 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:20:25.405513+0000", 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.2572999999999999e-05 2024-08-06T18:29:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:575,from:0,reservation_nonce:0)", 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 484.164870974, 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00039962, 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578021+0000", 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578023+0000", 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.736e-06 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578030+0000", 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.0140000000000001e-06 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578056+0000", 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6302999999999999e-05 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578093+0000", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6529000000000002e-05 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578174+0000", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.1209000000000001e-05 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578203+0000", 2024-08-06T18:29:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9045999999999999e-05 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.578421+0000", 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021778300000000001 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:29:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 484.16345701, 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00030426500000000002, 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579435+0000", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579437+0000", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.734e-06 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579490+0000", 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.3355e-05 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579515+0000", 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.5236999999999999e-05 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579527+0000", 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1482e-05 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579637+0000", 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011032200000000001 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579666+0000", 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9186000000000001e-05 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:21:47.579739+0000", 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.2948999999999996e-05 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:599,from:0,reservation_nonce:0)", 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 383.96052187399999, 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188605, 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782371+0000", 2024-08-06T18:29:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8299999999999995e-07 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782375+0000", 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7780000000000001e-06 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782384+0000", 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.986e-06 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782392+0000", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6650000000000003e-06 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782430+0000", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8778e-05 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782444+0000", 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3981999999999999e-05 2024-08-06T18:29:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782559+0000", 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011443299999999999 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 383.959873593, 2024-08-06T18:29:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5184000000000005e-05, 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:29:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783019+0000", 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7199999999999999e-07 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783022+0000", 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4340000000000001e-06 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783029+0000", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6590000000000001e-06 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783032+0000", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2650000000000001e-06 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783061+0000", 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8751999999999998e-05 2024-08-06T18:29:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783067+0000", 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783084+0000", 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:649,from:0,reservation_nonce:0)", 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.64384945800001, 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021522199999999999, 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099044+0000", 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.719e-06 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099049+0000", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4410000000000003e-06 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099061+0000", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2422999999999999e-05 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099070+0000", 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6349999999999995e-06 2024-08-06T18:29:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099113+0000", 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3393999999999998e-05 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099128+0000", 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5233e-05 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099258+0000", 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012937699999999999 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 215.64301265700001, 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4050999999999999e-05, 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099880+0000", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5199999999999996e-07 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099886+0000", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1040000000000003e-06 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099896+0000", 2024-08-06T18:29:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8789999999999998e-06 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099901+0000", 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970000000000003e-06 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099938+0000", 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6542999999999999e-05 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.794 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099944+0000", 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099963+0000", 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998e-05 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:699,from:0,reservation_nonce:0)", 2024-08-06T18:29:51.795 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.449473894, 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026914199999999999, 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.796 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293420+0000", 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9920000000000002e-06 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293427+0000", 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6719999999999998e-06 2024-08-06T18:29:51.797 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293442+0000", 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5124e-05 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293453+0000", 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.798 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293534+0000", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0717999999999995e-05 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293552+0000", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7982000000000002e-05 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293687+0000", 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013563200000000001 2024-08-06T18:29:51.799 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 40.448666473000003, 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9434999999999999e-05, 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:29:51.800 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:29:51.801 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000002e-07 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294229+0000", 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9280000000000002e-06 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294255+0000", 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6409000000000001e-05 2024-08-06T18:29:51.802 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294261+0000", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6300000000000003e-06 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294298+0000", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7233000000000001e-05 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294304+0000", 2024-08-06T18:29:51.803 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7960000000000001e-06 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294325+0000", 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0930000000000001e-05 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:29:51.804 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:29:51.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: pgmap v2100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:29:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:29:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: osdmap e710: 12 total, 3 up, 11 in 2024-08-06T18:29:51.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:51 smithi107 ceph-mon[32381]: osdmap e711: 12 total, 3 up, 11 in 2024-08-06T18:29:51.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:52.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:52.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: pgmap v2100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: osdmap e710: 12 total, 3 up, 11 in 2024-08-06T18:29:52.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:51 smithi086 ceph-mon[29229]: osdmap e711: 12 total, 3 up, 11 in 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: pgmap v2100: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3428101721' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: osdmap e710: 12 total, 3 up, 11 in 2024-08-06T18:29:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:51 smithi117 ceph-mon[32835]: osdmap e711: 12 total, 3 up, 11 in 2024-08-06T18:29:52.235 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:52.366 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:52.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:29:52.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:52.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:52.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:52.634 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:52.638 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:52.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:52.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:52.921 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:52.921 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:53.034 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:53.160 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:53.160 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:53.160 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:53.160 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:53.160 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:53.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:53.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:53.161 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:53.166 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:53.166 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:53.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:53.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:53.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:53.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:53.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:53.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:29:53.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:53.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:53.455 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:29:53.456 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:29:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:29:53.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:29:53.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:53.816 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:29:53.816 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:29:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:29:53.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:53.950 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:53.953 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:53.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:54.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:53 smithi086 ceph-mon[29229]: pgmap v2103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/801147042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:53 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:53 smithi107 ceph-mon[32381]: pgmap v2103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:54.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/801147042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:54.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:53 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:53 smithi117 ceph-mon[32835]: pgmap v2103: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/801147042' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:53 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:54.142 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:54.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:29:54.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:54.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:54.399 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:54.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:54.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:54.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:54.775 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:54.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:29:54.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:55.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:55.072 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:55.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:55.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:55.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:55.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:55.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:55.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:55.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:55.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:55.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:55.955 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:29:55.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:56.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:55 smithi086 ceph-mon[29229]: pgmap v2104: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:55 smithi117 ceph-mon[32835]: pgmap v2104: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:56.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:56.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:55 smithi107 ceph-mon[32381]: pgmap v2104: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:56.229 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:56.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:56.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:56.528 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:56.662 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:56.730 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:56.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:29:56.830 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:29:56.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:57.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:57.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:57.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:29:57.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:57.327 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:57.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:29:57.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:57.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:57.589 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:29:57.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:29:57.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:29:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: pgmap v2105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: osdmap e712: 12 total, 3 up, 11 in 2024-08-06T18:29:57.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:29:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:29:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:29:57.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:29:57.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:58.001 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: pgmap v2105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: osdmap e712: 12 total, 3 up, 11 in 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:29:58.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:29:58.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: pgmap v2105: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2842368154' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: osdmap e712: 12 total, 3 up, 11 in 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:29:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:29:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:29:58.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:29:58.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:29:58.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:58.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:58.639 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:29:58.639 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:29:58.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:29:58.647 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:29:58.647 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:29:58.647 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:29:58.647 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:29:58.648 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:29:58.665 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:29:58.665 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:29:58.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:29:58.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:29:58.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:29:58.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:29:58.666 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:29:58.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:58.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:29:58.916 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1071217966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:59.046 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:59.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1071217966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1071217966' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:29:59.135 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:29:59.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:29:59.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:29:59.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:29:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:29:59.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:29:59.428 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:29:59.429 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:29:59.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:59.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:29:59.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:29:59.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:29:59.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:29:59.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:29:59.902 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:29:59.938 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:59 smithi107 ceph-mon[32381]: pgmap v2107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:29:59.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:59.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:59.939 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:29:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:29:59.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:00.007 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:59 smithi086 ceph-mon[29229]: pgmap v2107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.010 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.011 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:29:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:59 smithi117 ceph-mon[32835]: pgmap v2107: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:29:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:00.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:00.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:00.337 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:00.338 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:00.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:00.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:00.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:00.814 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:00.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:01.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:01.066 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:30:01.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 1.0 is stuck stale for 39m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.2 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck inactive for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.6 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.8 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.9 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.a is stuck stale for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.c is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.d is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.e is stuck inactive for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.f is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.10 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.13 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.15 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.18 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.0 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.3 is stuck undersized for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.4 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.5 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.7 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.a is stuck undersized for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.c is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.e is stuck undersized for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.11 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.071 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.12 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.14 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.16 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.17 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1a is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1c is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1d is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1e is stuck undersized for 38m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.072 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:00 smithi086 ceph-mon[29229]: pg 2.1f is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:01.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:30:01.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:30:01.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 1.0 is stuck stale for 39m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.2 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck inactive for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.6 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.8 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.9 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.a is stuck stale for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.c is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.d is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.e is stuck inactive for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.f is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.10 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.13 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.15 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.18 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.115 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.0 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.3 is stuck undersized for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.4 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.5 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.7 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.a is stuck undersized for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.c is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.e is stuck undersized for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.11 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.116 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.12 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.14 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.16 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.17 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1a is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1c is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1d is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1e is stuck undersized for 38m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:00 smithi117 ceph-mon[32835]: pg 2.1f is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.212 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: Health detail: HEALTH_WARN noscrub,nodeep-scrub flag(s) set; 8 osds down; 1 host (4 osds) down; Reduced data availability: 30 pgs inactive, 6 pgs stale; Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: [WRN] OSDMAP_FLAGS: noscrub,nodeep-scrub flag(s) set 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: [WRN] OSD_DOWN: 8 osds down 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.1 (root=default,host=smithi086) is down 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.4 (root=default,host=smithi107) is down 2024-08-06T18:30:01.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.5 (root=default,host=smithi107) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.7 (root=default,host=smithi107) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.8 (root=default,host=smithi117) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.9 (root=default,host=smithi117) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.10 (root=default,host=smithi117) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: osd.11 (root=default,host=smithi117) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: [WRN] OSD_HOST_DOWN: 1 host (4 osds) down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: host smithi117 (root=default) (4 osds) is down 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: [WRN] PG_AVAILABILITY: Reduced data availability: 30 pgs inactive, 6 pgs stale 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 1.0 is stuck stale for 39m, current state stale+active+clean, last acting [7,11] 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.2 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck inactive for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.6 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.8 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.9 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.a is stuck stale for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.c is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.d is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.e is stuck inactive for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.f is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.215 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.10 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.13 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.216 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.15 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.18 is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck inactive for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1b is stuck inactive for 39m, current state unknown, last acting [] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck stale for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck inactive for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: [WRN] PG_DEGRADED: Degraded data redundancy: 18 pgs undersized 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.0 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.217 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.3 is stuck undersized for 39m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.4 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.5 is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.7 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.a is stuck undersized for 38m, current state stale+undersized+peered, last acting [3] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.c is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.e is stuck undersized for 38m, current state undersized+peered, last acting [2] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.11 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.12 is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.14 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.16 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.17 is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1a is stuck undersized for 39m, current state undersized+peered, last acting [6] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1c is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.218 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1d is stuck undersized for 39m, current state stale+undersized+peered, last acting [10] 2024-08-06T18:30:01.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1e is stuck undersized for 38m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.219 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:00 smithi107 ceph-mon[32381]: pg 2.1f is stuck undersized for 39m, current state undersized+peered, last acting [0] 2024-08-06T18:30:01.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:01.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:01.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:01.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:01.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:01.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:30:01.474 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:01.557 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:01.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:01.691 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:01.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:01.803 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:30:01.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:01 smithi107 ceph-mon[32381]: pgmap v2108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:01 smithi107 ceph-mon[32381]: osdmap e713: 12 total, 3 up, 11 in 2024-08-06T18:30:02.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:02.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:01 smithi117 ceph-mon[32835]: pgmap v2108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:02.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:02.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:02.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:01 smithi117 ceph-mon[32835]: osdmap e713: 12 total, 3 up, 11 in 2024-08-06T18:30:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:01 smithi086 ceph-mon[29229]: pgmap v2108: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2034912112' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:02.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:01 smithi086 ceph-mon[29229]: osdmap e713: 12 total, 3 up, 11 in 2024-08-06T18:30:02.136 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:02.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:30:02.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:02.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:02.382 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:02.402 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:02.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:02.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:02.555 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:02.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:02.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:02.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:03.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:03.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:03.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:03.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:03.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:03.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:03.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:03.987 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:04.001 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:04.043 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:03 smithi086 ceph-mon[29229]: pgmap v2110: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:04.043 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:04.043 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:04.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:03 smithi117 ceph-mon[32835]: pgmap v2110: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:04.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:04.141 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:04.141 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:04.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:04.158 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:04.159 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:04.159 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:04.159 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:04.159 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:04.159 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:03 smithi107 ceph-mon[32381]: pgmap v2110: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:04.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:04.160 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:04.175 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:04.177 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:04.177 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:04.177 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:04.177 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:04.177 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:04.178 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:04.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:04.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:04.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:04.608 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:04.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:30:04.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:04.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:04.856 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:04.878 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:04.879 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:04.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:05.009 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:30:05.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:05.085 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2149616977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:05.085 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.086 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:04 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.086 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:05.086 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:05.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2149616977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:05.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:04 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2149616977' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:04 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:05.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:05.263 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:05.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:05.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:05.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:05.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:05.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:05.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:06.016 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:06.017 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:05 smithi107 ceph-mon[32381]: pgmap v2111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:06.017 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:06.017 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:06.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:05 smithi086 ceph-mon[29229]: pgmap v2111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:06.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:06.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:05 smithi117 ceph-mon[32835]: pgmap v2111: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:06.125 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:06.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:06.374 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:06.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:06.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:06.661 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:06.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:06.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:30:06.895 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:06.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:30:06.933 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:06.955 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:06 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:06.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:06.955 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:06.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:06 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:07.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:06 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:07.137 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:07.144 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:07.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:07.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:07.463 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:07.485 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:07.485 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:07.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:07.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:07.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:07.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:07.855 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:07.862 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:30:07.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:08.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:07 smithi086 ceph-mon[29229]: pgmap v2112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:08.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:08.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:08.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:08.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:07 smithi117 ceph-mon[32835]: pgmap v2112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:08.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:07 smithi107 ceph-mon[32381]: pgmap v2112: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:08.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:08.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:08.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:08.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:08.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:08.514 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:08.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:08.646 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:08.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:08.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:08.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:08 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:08.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:08.967 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:08 smithi086 ceph-mon[29229]: osdmap e714: 12 total, 3 up, 11 in 2024-08-06T18:30:08.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:08.967 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:09.042 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:09.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:08 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:09.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:09.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:08 smithi117 ceph-mon[32835]: osdmap e714: 12 total, 3 up, 11 in 2024-08-06T18:30:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:09.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:30:09.175 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:08 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/148793794' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:09.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:08 smithi107 ceph-mon[32381]: osdmap e714: 12 total, 3 up, 11 in 2024-08-06T18:30:09.261 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:09.261 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:09.306 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:09.329 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:09.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:09.505 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:09.506 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:09.506 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:09.523 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:09.524 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:09.541 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:09.542 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:09.543 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:09.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:09.702 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:09.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:09.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:09.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:09.963 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:10.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:09 smithi117 ceph-mon[32835]: pgmap v2114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:10.127 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4078687848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:09 smithi107 ceph-mon[32381]: pgmap v2114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:10.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4078687848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:10.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:10.300 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:09 smithi086 ceph-mon[29229]: pgmap v2114: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:10.300 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4078687848' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:10.330 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:10.330 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:10.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:10.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:10.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:30:10.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:10.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:10.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:10.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:10.793 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:10.827 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:10.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:11.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:11.040 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.040 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.040 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:11.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:11.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:11.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:11.414 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:11.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:30:11.546 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:11.660 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:11.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:11.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:11.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:11.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:11 smithi086 ceph-mon[29229]: pgmap v2115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:11.962 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:11 smithi086 ceph-mon[29229]: osdmap e715: 12 total, 3 up, 11 in 2024-08-06T18:30:12.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:11 smithi117 ceph-mon[32835]: pgmap v2115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:11 smithi117 ceph-mon[32835]: osdmap e715: 12 total, 3 up, 11 in 2024-08-06T18:30:12.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:11 smithi107 ceph-mon[32381]: pgmap v2115: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:11 smithi107 ceph-mon[32381]: osdmap e715: 12 total, 3 up, 11 in 2024-08-06T18:30:12.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:30:12.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:12.465 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:12.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:12.604 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:12.729 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:12.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:12.889 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:30:12.999 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:13.059 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:13.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:13.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:13.262 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:13.393 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:13.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:13.578 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:13.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:30:13.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:13.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:13.791 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:13.828 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:13.923 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:14.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:14.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:13 smithi117 ceph-mon[32835]: pgmap v2117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:14.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:13 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:14.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:14.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:13 smithi117 ceph-mon[32835]: osdmap e716: 12 total, 3 up, 11 in 2024-08-06T18:30:14.190 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:13 smithi107 ceph-mon[32381]: pgmap v2117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:13 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:13 smithi107 ceph-mon[32381]: osdmap e716: 12 total, 3 up, 11 in 2024-08-06T18:30:14.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:14.347 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:13 smithi086 ceph-mon[29229]: pgmap v2117: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:14.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:13 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:14.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/683779143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:14.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:13 smithi086 ceph-mon[29229]: osdmap e716: 12 total, 3 up, 11 in 2024-08-06T18:30:14.348 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:14.348 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:14.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:14.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:14.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:14.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:14.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:14.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:14.917 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:14.917 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:14.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:15.015 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:15.016 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:15.016 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:15.032 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:15.033 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:15.033 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:15.033 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:15.033 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:15.050 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:15.051 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:15.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:15.236 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:15.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:15.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:15.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:15.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:15.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:15.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:30:15.733 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:15.733 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:15.773 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:15.774 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:30:15.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:15.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:15.974 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:16.023 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:16.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:16.096 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:16.096 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:16.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:15 smithi086 ceph-mon[29229]: pgmap v2119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:16.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/807829403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:16.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:15 smithi117 ceph-mon[32835]: pgmap v2119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:16.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/807829403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:16.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:15 smithi107 ceph-mon[32381]: pgmap v2119: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/807829403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:16.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:16.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:16.347 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:16.348 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:16.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:16.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:16.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:16.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:16.987 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:16 smithi117 ceph-mon[32835]: osdmap e717: 12 total, 3 up, 11 in 2024-08-06T18:30:17.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:17.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:17.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:16 smithi086 ceph-mon[29229]: osdmap e717: 12 total, 3 up, 11 in 2024-08-06T18:30:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:16 smithi107 ceph-mon[32381]: osdmap e717: 12 total, 3 up, 11 in 2024-08-06T18:30:17.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:17.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:17.426 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:17.426 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:17.441 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:17.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:17.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:17.715 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:17.715 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:17.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:30:17.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:17.905 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:30:17.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:18.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:17 smithi117 ceph-mon[32835]: pgmap v2121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:18.002 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:18.028 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:18.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:30:18.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:18.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:17 smithi086 ceph-mon[29229]: pgmap v2121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:18.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:18.118 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:18.118 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:17 smithi107 ceph-mon[32381]: pgmap v2121: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:18.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:18.319 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:18.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:18.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:18.468 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:18.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:18.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:18.752 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:18.768 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:18.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:19.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:18 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:19.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:19.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:18 smithi117 ceph-mon[32835]: osdmap e718: 12 total, 3 up, 11 in 2024-08-06T18:30:19.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:18 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:30:19.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:18 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:30:19.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:19.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:18 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:19.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:18 smithi086 ceph-mon[29229]: osdmap e718: 12 total, 3 up, 11 in 2024-08-06T18:30:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:18 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:30:19.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:18 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:30:19.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:18 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2382724136' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:18 smithi107 ceph-mon[32381]: osdmap e718: 12 total, 3 up, 11 in 2024-08-06T18:30:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:18 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:30:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:18 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:30:19.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:19.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:19.446 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:19.446 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:19.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:19.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:19.754 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:19.816 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:19.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:20.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:20.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:19 smithi086 ceph-mon[29229]: pgmap v2123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:20.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:19 smithi086 ceph-mon[29229]: osdmap e719: 12 total, 3 up, 11 in 2024-08-06T18:30:20.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:20.085 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:19 smithi117 ceph-mon[32835]: pgmap v2123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:19 smithi117 ceph-mon[32835]: osdmap e719: 12 total, 3 up, 11 in 2024-08-06T18:30:20.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:19 smithi107 ceph-mon[32381]: pgmap v2123: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:20.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:19 smithi107 ceph-mon[32381]: osdmap e719: 12 total, 3 up, 11 in 2024-08-06T18:30:20.335 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:20.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:30:20.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:20.344 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:20.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:20.470 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:20.470 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:20.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:20.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:20.478 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:20.479 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:20.479 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:20.479 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:20.480 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:20.480 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:20.480 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:20.580 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:20.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:20.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:30:20.740 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:20.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:20.915 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:20.915 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:21.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:21.017 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2337291171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:21.018 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:21.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:21.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:21.084 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2337291171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:21.138 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:21.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2337291171' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:21.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:21.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:21.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:21.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:21.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:21.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:21.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:22.012 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:22.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:22.092 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:22.092 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:22.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:21 smithi086 ceph-mon[29229]: pgmap v2125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:22.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:21 smithi117 ceph-mon[32835]: pgmap v2125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:21 smithi107 ceph-mon[32381]: pgmap v2125: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:22.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:22.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:22.601 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:22.656 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:22.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:30:22.731 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:22.871 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:22.923 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:30:22.946 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:23.006 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:23.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:23.051 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:30:23.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:23.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:23.165 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:23.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:23.282 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:23.282 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:23.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:23.543 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:23.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:23.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:23.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:23.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:23.965 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:23 smithi117 ceph-mon[32835]: pgmap v2126: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:23.965 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:23 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:23.965 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:23.965 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:23 smithi117 ceph-mon[32835]: osdmap e720: 12 total, 3 up, 11 in 2024-08-06T18:30:23.984 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:23.984 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:23.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:23 smithi086 ceph-mon[29229]: pgmap v2126: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:23.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:23 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:23.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:23.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:23 smithi086 ceph-mon[29229]: osdmap e720: 12 total, 3 up, 11 in 2024-08-06T18:30:24.075 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:24.098 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:24.098 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:24.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:23 smithi107 ceph-mon[32381]: pgmap v2126: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:24.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:23 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:24.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/571493487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:24.099 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:23 smithi107 ceph-mon[32381]: osdmap e720: 12 total, 3 up, 11 in 2024-08-06T18:30:24.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:24.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:24.363 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:24.472 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:24.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:24.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:24.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:24.525 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:24.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:24.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:24.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:24.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:24.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:24.985 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:24.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:30:24.995 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:25.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:25.144 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:25.144 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:25.230 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:30:25.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:25.399 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:25.517 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:25.517 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:25.517 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:25.533 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:25.534 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:25.535 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:25.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:25.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:25.535 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:25.551 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:25.552 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:25.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:25.552 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:25.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:25.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:25.553 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:25.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:25.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:30:25.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:25.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:26.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:25 smithi086 ceph-mon[29229]: pgmap v2128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/919359373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:26.198 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:25 smithi117 ceph-mon[32835]: pgmap v2128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:26.198 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/919359373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:26.198 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:26.200 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:25 smithi107 ceph-mon[32381]: pgmap v2128: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 332 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/919359373' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:26.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:26.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:26.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:26.414 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:26.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:26.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:26.661 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:26.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:26.783 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:26.783 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:26.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:26.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:27.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:27.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:26 smithi086 ceph-mon[29229]: osdmap e721: 12 total, 3 up, 11 in 2024-08-06T18:30:27.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:27.188 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.188 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.188 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:26 smithi117 ceph-mon[32835]: osdmap e721: 12 total, 3 up, 11 in 2024-08-06T18:30:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:26 smithi107 ceph-mon[32381]: osdmap e721: 12 total, 3 up, 11 in 2024-08-06T18:30:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:27.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:30:27.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:27.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:27.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:27.653 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:27.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:27.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:28.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:28.146 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:28.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:28.199 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:27 smithi117 ceph-mon[32835]: pgmap v2129: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:28.200 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:27 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:30:28.200 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:27 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:30:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:27 smithi107 ceph-mon[32381]: pgmap v2129: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:27 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:30:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:27 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:30:28.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:27 smithi086 ceph-mon[29229]: pgmap v2129: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:28.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:27 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:30:28.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:27 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:30:28.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:28.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:30:28.438 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:28.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:28.704 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:28.730 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:28.730 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:28.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:28.933 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:30:28.978 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:29.002 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:29.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:29.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:29.298 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:29.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:29.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:29.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:29.602 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:29.602 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:29.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:29.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:29.825 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:29.915 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:29.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:29.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:30.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:29 smithi117 ceph-mon[32835]: pgmap v2131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:29 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:29 smithi117 ceph-mon[32835]: osdmap e722: 12 total, 3 up, 11 in 2024-08-06T18:30:30.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:30.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:29 smithi107 ceph-mon[32381]: pgmap v2131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:30.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:29 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:30.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:30.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:29 smithi107 ceph-mon[32381]: osdmap e722: 12 total, 3 up, 11 in 2024-08-06T18:30:30.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:30.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:29 smithi086 ceph-mon[29229]: pgmap v2131: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:30.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:29 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:30:30.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3409314708' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:30.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:29 smithi086 ceph-mon[29229]: osdmap e722: 12 total, 3 up, 11 in 2024-08-06T18:30:30.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:30.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:30.440 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:30.441 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:30:30.451 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:30.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:30.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:30.853 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:30.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:31.067 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:31.067 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:31.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:31.127 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:31.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:31.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:31.526 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:31.661 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:31.746 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:31.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:31.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:31.972 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:31 smithi117 ceph-mon[32835]: pgmap v2133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:31.972 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:31 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:30:31.972 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:31 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:30:31.972 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:31 smithi117 ceph-mon[32835]: osdmap e723: 12 total, 3 up, 11 in 2024-08-06T18:30:32.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:32.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:32.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:31 smithi086 ceph-mon[29229]: pgmap v2133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:32.206 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:31 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:30:32.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:31 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:30:32.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:31 smithi086 ceph-mon[29229]: osdmap e723: 12 total, 3 up, 11 in 2024-08-06T18:30:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:31 smithi107 ceph-mon[32381]: pgmap v2133: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:31 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:30:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:31 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:30:32.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:31 smithi107 ceph-mon[32381]: osdmap e723: 12 total, 3 up, 11 in 2024-08-06T18:30:32.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:32.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:32.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:32.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:32.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:32.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:33.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:30:33.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:33.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:33.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:33.464 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:33.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:33.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:33.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:33.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:33.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:33.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:33.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:33.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:33.939 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:33.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:33.970 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:33.970 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:33.970 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:33.988 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:33.989 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:33.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:33.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:33.989 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:34.006 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:34.006 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:34.006 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:34.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:34.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:34.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:34.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:34.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:34.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:33 smithi117 ceph-mon[32835]: pgmap v2135: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:34.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:33 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:30:34.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:33 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:30:34.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:33 smithi117 ceph-mon[32835]: osdmap e724: 12 total, 3 up, 11 in 2024-08-06T18:30:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:33 smithi107 ceph-mon[32381]: pgmap v2135: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:33 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:30:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:33 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:30:34.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:33 smithi107 ceph-mon[32381]: osdmap e724: 12 total, 3 up, 11 in 2024-08-06T18:30:34.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:34.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:34.239 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:34.239 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:33 smithi086 ceph-mon[29229]: pgmap v2135: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:34.239 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:33 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:30:34.240 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:33 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:30:34.240 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:33 smithi086 ceph-mon[29229]: osdmap e724: 12 total, 3 up, 11 in 2024-08-06T18:30:34.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:34.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:34.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:34.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:34.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:34.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:34.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:35.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:35.042 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/409560033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:35.042 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.042 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.042 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:35.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/409560033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:35.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.287 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/409560033' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:35.385 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:35.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:35.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:35.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:35.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:35.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:35.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:35.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:35.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:35 smithi086 ceph-mon[29229]: pgmap v2137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:35.981 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:35.981 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:36.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:35 smithi117 ceph-mon[32835]: pgmap v2137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:36.167 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:36.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:30:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:35 smithi107 ceph-mon[32381]: pgmap v2137: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:36.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:36.266 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:36.267 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:36.267 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:36.267 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:36.357 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:36.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:36.617 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:36.618 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:36.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:36.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:36.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:37.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:37.031 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:37.159 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:37.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:37.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:37.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:37.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:37.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:37.698 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:37.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:37.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:37 smithi107 ceph-mon[32381]: pgmap v2138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:37.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:37 smithi107 ceph-mon[32381]: osdmap e725: 12 total, 3 up, 11 in 2024-08-06T18:30:38.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:37 smithi086 ceph-mon[29229]: pgmap v2138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:37 smithi086 ceph-mon[29229]: osdmap e725: 12 total, 3 up, 11 in 2024-08-06T18:30:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:37 smithi117 ceph-mon[32835]: pgmap v2138: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:37 smithi117 ceph-mon[32835]: osdmap e725: 12 total, 3 up, 11 in 2024-08-06T18:30:38.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:38.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:38.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:38.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:30:38.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:38.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:38.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:38.996 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:30:39.019 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:39.019 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:39.040 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:39.176 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:39.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:39.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:39.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:39.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:39.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:39.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:39.713 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:39.713 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:30:39.740 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:30:39.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:39.873 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:39 smithi107 ceph-mon[32381]: pgmap v2140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:39.970 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:39.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:40.008 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:39 smithi117 ceph-mon[32835]: pgmap v2140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:40.093 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:39 smithi086 ceph-mon[29229]: pgmap v2140: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:40.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:40.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:40.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:40.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:40.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:40.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:40.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:40.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:40.841 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:30:40.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:40.937 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:41.063 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:41.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:41.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:41.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:30:41.274 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:41.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:41.335 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:41.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:41.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:41.608 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:41.608 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:41.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:41.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:41.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:41 smithi086 ceph-mon[29229]: pgmap v2141: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:41 smithi086 ceph-mon[29229]: osdmap e726: 12 total, 3 up, 11 in 2024-08-06T18:30:41.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:41 smithi086 ceph-mon[29229]: osdmap e727: 12 total, 3 up, 11 in 2024-08-06T18:30:41.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:41.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:41 smithi117 ceph-mon[32835]: pgmap v2141: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:41.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:41.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:41.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:41 smithi117 ceph-mon[32835]: osdmap e726: 12 total, 3 up, 11 in 2024-08-06T18:30:41.901 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:41 smithi117 ceph-mon[32835]: osdmap e727: 12 total, 3 up, 11 in 2024-08-06T18:30:41.902 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:42.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:42.114 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:42.114 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:42.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:42.134 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:42.134 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:42.134 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:42.135 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:42.135 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:42.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:42.152 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:42.153 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:42.153 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:42.153 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:42.153 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:42.153 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:42.156 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:41 smithi107 ceph-mon[32381]: pgmap v2141: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1021378815' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:30:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:41 smithi107 ceph-mon[32381]: osdmap e726: 12 total, 3 up, 11 in 2024-08-06T18:30:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:41 smithi107 ceph-mon[32381]: osdmap e727: 12 total, 3 up, 11 in 2024-08-06T18:30:42.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:42.409 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:42.482 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:42.482 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:42.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:42.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:42.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:42.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:43.062 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:43.062 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:43.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1871399249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:43.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:42 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1871399249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:42 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:43.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1871399249' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:43.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:42 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:43.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:43.507 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:43.507 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:43.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:43.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:30:43.659 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:43.660 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:43.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:43.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:43.940 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.941 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:43 smithi086 ceph-mon[29229]: pgmap v2144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:43 smithi107 ceph-mon[32381]: pgmap v2144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:44.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:44.019 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:44.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:43 smithi117 ceph-mon[32835]: pgmap v2144: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:44.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:44.219 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:44.219 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:44.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:44.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:44.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:44.663 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:30:44.663 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:30:44.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:44.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:44.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:45.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:45.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:45.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:45.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:45.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:45.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:45.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:30:45.633 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:45.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:45.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:45.852 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:45.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:30:45.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:45.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:45 smithi107 ceph-mon[32381]: pgmap v2145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:45.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:45.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:46.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:46.080 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:45 smithi117 ceph-mon[32835]: pgmap v2145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:45 smithi086 ceph-mon[29229]: pgmap v2145: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:46.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:46.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:46.360 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:46.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:46.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:46.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:46.842 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:46.843 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:46.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:46.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:47.122 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:47.255 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:47.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:47.455 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:47.455 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:47.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:47.654 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:47.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:47.855 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:30:47.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:48.062 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:48.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:48.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:48.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:47 smithi086 ceph-mon[29229]: pgmap v2146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:47 smithi117 ceph-mon[32835]: pgmap v2146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:48.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:47 smithi107 ceph-mon[32381]: pgmap v2146: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:30:48.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:48.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:48.490 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:48.491 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:48.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:48.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:48.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:49.021 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:49.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:48 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:49.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:49.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:48 smithi086 ceph-mon[29229]: osdmap e728: 12 total, 3 up, 11 in 2024-08-06T18:30:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:48 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:49.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:48 smithi117 ceph-mon[32835]: osdmap e728: 12 total, 3 up, 11 in 2024-08-06T18:30:49.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:49.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:48 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:49.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/145158083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:30:49.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:48 smithi107 ceph-mon[32381]: osdmap e728: 12 total, 3 up, 11 in 2024-08-06T18:30:49.213 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:49.213 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:49.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:49.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:49.242 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:49.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:49.251 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:49.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:49.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:49.252 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:49.252 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:49.252 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:49.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:49.418 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:49.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:49.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:49.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:49.951 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:49 smithi107 ceph-mon[32381]: pgmap v2148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4155822456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:49.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:50.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:49 smithi086 ceph-mon[29229]: pgmap v2148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4155822456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:50.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:49 smithi117 ceph-mon[32835]: pgmap v2148: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4155822456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:50.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:50.220 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:50.264 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:50.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:50.354 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:50.380 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:50.380 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:50.380 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:50.380 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:50.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:50.491 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:50.491 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:50.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:50.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:50.892 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:30:50.918 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:51.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:30:51.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:51.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:51.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:51.157 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:51.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:51.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:51.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:51.623 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:30:51.623 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:30:51.623 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:30:51.623 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:30:51.623 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:30:51.682 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:51.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:51.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:52.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:51 smithi107 ceph-mon[32381]: pgmap v2149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:51 smithi107 ceph-mon[32381]: osdmap e729: 12 total, 3 up, 11 in 2024-08-06T18:30:52.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:52.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:51 smithi117 ceph-mon[32835]: pgmap v2149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:52.235 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:51 smithi117 ceph-mon[32835]: osdmap e729: 12 total, 3 up, 11 in 2024-08-06T18:30:52.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:30:52.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:51 smithi086 ceph-mon[29229]: pgmap v2149: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:52.288 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:51 smithi086 ceph-mon[29229]: osdmap e729: 12 total, 3 up, 11 in 2024-08-06T18:30:52.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:52.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:52.325 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:52.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:52.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:30:52.479 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:52.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:52.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:52.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:52.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:52.872 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:53.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:53.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:53.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:53.407 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:53.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:53.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:53.565 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:53.604 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:53.610 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:53.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:53.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:53.821 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:53.821 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:53.946 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:53.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:53 smithi107 ceph-mon[32381]: pgmap v2151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:53.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:53.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:54.075 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:53 smithi086 ceph-mon[29229]: pgmap v2151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:54.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:54.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:54.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:53 smithi117 ceph-mon[32835]: pgmap v2151: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:54.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:54.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:54.626 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:54.626 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:54.626 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:54.626 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:54.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:54.881 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:54.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:54.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:54.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:54.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:55.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:55.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:55.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:55.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:54 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:55.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:55.417 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:55.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:30:55.618 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:55.618 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:55.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:55.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:55.894 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:55.894 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:55.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:30:55.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:30:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:30:55.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:56.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:56.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:55 smithi086 ceph-mon[29229]: pgmap v2152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:55 smithi107 ceph-mon[32381]: pgmap v2152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:56.223 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:56.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:55 smithi117 ceph-mon[32835]: pgmap v2152: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:56.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:56.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:56.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:56.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:56.775 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:56.901 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:30:56.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:575,from:6,reservation_nonce:25)", 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 549.40485885199996, 2024-08-06T18:30:56.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00020588599999999999, 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577631+0000", 2024-08-06T18:30:56.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577634+0000", 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2400000000000002e-06 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577646+0000", 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2357e-05 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.983 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577680+0000", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.4119000000000003e-05 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577696+0000", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5716e-05 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577796+0000", 2024-08-06T18:30:56.984 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010001900000000001 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577811+0000", 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5299e-05 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.577837+0000", 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6135999999999998e-05 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:56.985 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:575/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 549.40371035199996, 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00028445800000000002, 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.986 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578780+0000", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578786+0000", 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5400000000000001e-06 2024-08-06T18:30:56.987 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578798+0000", 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1236000000000001e-05 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578827+0000", 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.9070000000000001e-05 2024-08-06T18:30:56.988 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578842+0000", 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4916000000000001e-05 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578978+0000", 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136534 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.989 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.578998+0000", 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9412999999999999e-05 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.579064+0000", 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6748999999999994e-05 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.990 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:575,from:6,reservation_nonce:0)", 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 549.40239318500005, 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016099300000000001, 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:56.991 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580097+0000", 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580098+0000", 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3300000000000002e-07 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580101+0000", 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6940000000000001e-06 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580130+0000", 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8833e-05 2024-08-06T18:30:56.993 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580136+0000", 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7520000000000002e-06 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.994 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580238+0000", 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102543 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580249+0000", 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0823e-05 2024-08-06T18:30:56.995 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:21:47.580258+0000", 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.8149999999999995e-06 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:599,from:6,reservation_nonce:26)", 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 449.200466248, 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018547099999999999, 2024-08-06T18:30:56.996 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:56.997 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782025+0000", 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5340000000000001e-06 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:56.998 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782035+0000", 2024-08-06T18:30:56.999 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999993e-06 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782055+0000", 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0650999999999999e-05 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782064+0000", 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9209999999999998e-06 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.000 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782156+0000", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1822000000000003e-05 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782172+0000", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5611999999999999e-05 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782209+0000", 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7830999999999997e-05 2024-08-06T18:30:57.001 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:599/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 449.199824896, 2024-08-06T18:30:57.002 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111396, 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:30:57.003 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782666+0000", 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6899999999999998e-07 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782670+0000", 2024-08-06T18:30:57.004 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.048e-06 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782697+0000", 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6948999999999999e-05 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782702+0000", 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8199999999999996e-06 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.005 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782739+0000", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7166999999999998e-05 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782745+0000", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1430000000000003e-06 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782777+0000", 2024-08-06T18:30:57.006 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.18e-05 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:599,from:6,reservation_nonce:0)", 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 449.199131862, 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136124, 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.007 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783359+0000", 2024-08-06T18:30:57.008 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1699999999999999e-07 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783361+0000", 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5359999999999999e-06 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783367+0000", 2024-08-06T18:30:57.009 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.285e-06 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783376+0000", 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4780000000000008e-06 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783463+0000", 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6978999999999995e-05 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.010 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783480+0000", 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6634e-05 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783494+0000", 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4895000000000001e-05 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.011 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:649,from:6,reservation_nonce:28)", 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 280.88372343999998, 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147861, 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.012 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.013 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098768+0000", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4e-07 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098773+0000", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9139999999999999e-06 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098783+0000", 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0508e-05 2024-08-06T18:30:57.014 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098790+0000", 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5409999999999997e-06 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098871+0000", 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1824999999999997e-05 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.015 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098887+0000", 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5617e-05 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098915+0000", 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7515999999999999e-05 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.016 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:649/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 280.88297577899999, 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013723899999999999, 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.017 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099515+0000", 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6300000000000005e-07 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.018 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099522+0000", 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0110000000000003e-06 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099533+0000", 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1272e-05 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.019 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099540+0000", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4670000000000001e-06 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099619+0000", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9609000000000003e-05 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099628+0000", 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5110000000000003e-06 2024-08-06T18:30:57.020 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099652+0000", 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3706e-05 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.021 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:673,from:2,reservation_nonce:27)", 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 187.55212771800001, 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145129, 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:30:57.022 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430364+0000", 2024-08-06T18:30:57.023 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.513e-06 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430371+0000", 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0190000000000001e-06 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430385+0000", 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.024 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430396+0000", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0229e-05 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430466+0000", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0858000000000005e-05 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430481+0000", 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4596999999999999e-05 2024-08-06T18:30:57.025 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430508+0000", 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:673/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:30:57.026 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 187.55151997499999, 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111407, 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.027 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430971+0000", 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6599999999999996e-07 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.028 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430975+0000", 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2889999999999999e-06 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430986+0000", 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0729e-05 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.029 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430992+0000", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.694e-06 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431040+0000", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8418999999999998e-05 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431050+0000", 2024-08-06T18:30:57.030 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431082+0000", 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1754999999999997e-05 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.031 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:673,from:2,reservation_nonce:0)", 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 187.550906747, 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012740599999999999, 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:30:57.032 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431584+0000", 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.36e-07 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.033 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431586+0000", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0080000000000001e-06 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431590+0000", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0090000000000001e-06 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431596+0000", 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2759999999999997e-06 2024-08-06T18:30:57.034 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431681+0000", 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4974000000000003e-05 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431696+0000", 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4654000000000001e-05 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.035 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431711+0000", 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5248999999999999e-05 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:697,from:2,reservation_nonce:28)", 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 111.759039417, 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014356300000000001, 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.036 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.037 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223452+0000", 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3510000000000001e-06 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223460+0000", 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9049999999999997e-06 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223475+0000", 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5021999999999999e-05 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.038 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223483+0000", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0350000000000001e-06 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223540+0000", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7235999999999999e-05 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223561+0000", 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0309000000000002e-05 2024-08-06T18:30:57.039 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223594+0000", 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3704999999999997e-05 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.040 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:697,from:2,reservation_nonce:0)", 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 111.757866042, 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110475, 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:30:57.041 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.77e-07 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.042 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.043 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224627+0000", 2024-08-06T18:30:57.045 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.424e-06 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224633+0000", 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2169999999999999e-06 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224638+0000", 2024-08-06T18:30:57.046 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0010000000000001e-06 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224712+0000", 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4124999999999999e-05 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224723+0000", 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0895e-05 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.047 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224735+0000", 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1436e-05 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:699,from:6,reservation_nonce:30)", 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.68941187999999, 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022859099999999999, 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.048 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:30:57.049 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293081+0000", 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4099999999999998e-06 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293094+0000", 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2809999999999999e-05 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.050 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293125+0000", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0998999999999999e-05 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293139+0000", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4667e-05 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293256+0000", 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116816 2024-08-06T18:30:57.051 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293272+0000", 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5574e-05 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293307+0000", 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5315000000000003e-05 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.052 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:699,from:6,reservation_nonce:0)", 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 105.687803226, 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016018, 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.053 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7e-07 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294707+0000", 2024-08-06T18:30:57.054 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9466e-05 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294716+0000", 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6660000000000003e-06 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294722+0000", 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6489999999999996e-06 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:30:57.055 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294825+0000", 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102699 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294838+0000", 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3339999999999999e-05 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294847+0000", 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9900000000000003e-06 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.056 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:30:57.057 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:30:57.057 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:30:57.057 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:30:57.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:57.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:57.188 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:57.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:30:57.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:30:57.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:30:57.260 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:30:57.261 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:30:57.262 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:30:57.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:57.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:57.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:57.606 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:57.606 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:57.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:57.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:57.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:57.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:58.117 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: pgmap v2153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:30:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:30:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: osdmap e730: 12 total, 3 up, 11 in 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1769098602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:57 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:30:58.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: pgmap v2153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:30:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:30:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:58.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: osdmap e730: 12 total, 3 up, 11 in 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1769098602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:57 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: pgmap v2153: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/363622925' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: osdmap e730: 12 total, 3 up, 11 in 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1769098602' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:30:58.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:30:58.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:57 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:30:58.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:58.265 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:30:58.265 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:30:58.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:58.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:58.557 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:58.557 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:58.639 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:58.771 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:30:58.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:30:58.904 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:59.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:30:59.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:59.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:59.181 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:30:59.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:30:59.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:30:59.560 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:30:59.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:30:59.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:30:59.836 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:30:59.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:30:59.944 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:30:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:30:59.944 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:30:59.944 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:30:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:30:59.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:00.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: pgmap v2155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:31:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:30:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: pgmap v2155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:31:00.262 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:30:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: pgmap v2155: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:31:00.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:30:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:31:00.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:00.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:00.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:00.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:00.879 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:00.895 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:01.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:01.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:01.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:01.430 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:01.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:31:01.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:01.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:01.800 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:01.827 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:01.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:02.051 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:02.096 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:01 smithi086 ceph-mon[29229]: pgmap v2156: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:02.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:01 smithi086 ceph-mon[29229]: osdmap e731: 12 total, 3 up, 11 in 2024-08-06T18:31:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:01 smithi107 ceph-mon[32381]: pgmap v2156: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:01 smithi107 ceph-mon[32381]: osdmap e731: 12 total, 3 up, 11 in 2024-08-06T18:31:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:02.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:02.226 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:02.249 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:01 smithi117 ceph-mon[32835]: pgmap v2156: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:02.250 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:01 smithi117 ceph-mon[32835]: osdmap e731: 12 total, 3 up, 11 in 2024-08-06T18:31:02.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:02.380 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:02.381 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:02.489 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:02.625 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:02.760 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:02.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:03.027 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:03.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:03.281 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:03.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:31:03.297 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:03.431 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:03.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:03.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:03.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:03.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:03.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:03 smithi117 ceph-mon[32835]: pgmap v2158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:04.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:04.154 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:03 smithi086 ceph-mon[29229]: pgmap v2158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:03 smithi107 ceph-mon[32381]: pgmap v2158: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:04.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:04.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:04.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:04.518 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:04.519 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:04.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:04.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:04.870 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:04.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:05.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:05.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:05.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:05.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:05.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:05.438 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:05.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:05.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:05.841 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:05.969 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:31:05.974 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:06.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:05 smithi117 ceph-mon[32835]: pgmap v2159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:06.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:06.267 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:05 smithi107 ceph-mon[32381]: pgmap v2159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:06.267 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:06.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:05 smithi086 ceph-mon[29229]: pgmap v2159: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:06.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:06.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:06.508 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:06.643 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:06.774 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:06.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:06.910 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:06.911 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:06.912 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:06.966 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:06.966 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:07.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:07.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:07.200 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: osdmap e732: 12 total, 3 up, 11 in 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1284948456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:06 smithi117 ceph-mon[32835]: osdmap e733: 12 total, 3 up, 11 in 2024-08-06T18:31:07.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:07.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:07.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:07.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: osdmap e732: 12 total, 3 up, 11 in 2024-08-06T18:31:07.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1284948456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:06 smithi086 ceph-mon[29229]: osdmap e733: 12 total, 3 up, 11 in 2024-08-06T18:31:07.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:07.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:07.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2365177192' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: osdmap e732: 12 total, 3 up, 11 in 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1284948456' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:07.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:06 smithi107 ceph-mon[32381]: osdmap e733: 12 total, 3 up, 11 in 2024-08-06T18:31:07.565 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:07.566 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:07.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:07.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:07.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:07.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:07.990 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:07 smithi086 ceph-mon[29229]: pgmap v2161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:07.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:07.990 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:07.990 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:07.991 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:08.100 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:08.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:08.246 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:08.246 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:08.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:08.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:07 smithi117 ceph-mon[32835]: pgmap v2161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:08.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:07 smithi107 ceph-mon[32381]: pgmap v2161: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:08.494 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:08.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:08.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:08.522 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:08.523 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:08.523 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:08.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:08.762 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:31:08.768 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:08.783 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:08.784 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:08.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:09.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:09.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:09.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:09.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:09.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:09.260 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:31:09.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:09.346 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:09.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:09.567 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:31:09.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:09.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:09.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:09.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:10.002 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:09 smithi086 ceph-mon[29229]: pgmap v2163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:10.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:10.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:10.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:10.260 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:10.281 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:09 smithi107 ceph-mon[32381]: pgmap v2163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:10.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:09 smithi117 ceph-mon[32835]: pgmap v2163: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:10.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:10.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:10.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:10.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:10.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:11.057 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:11.058 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:11.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:11.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:11.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:11.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:11.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:11.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:11.870 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:11.977 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:31:12.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:12.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:11 smithi107 ceph-mon[32381]: pgmap v2164: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:12.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:12.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:11 smithi086 ceph-mon[29229]: pgmap v2164: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:12.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:11 smithi117 ceph-mon[32835]: pgmap v2164: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:12.401 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:12.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:12.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:12.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:12.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:12.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:12.667 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:12.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:12.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:13.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:13.205 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:13.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:13.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:12 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:13.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:13.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:12 smithi086 ceph-mon[29229]: osdmap e734: 12 total, 3 up, 11 in 2024-08-06T18:31:13.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:13.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:12 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:12 smithi117 ceph-mon[32835]: osdmap e734: 12 total, 3 up, 11 in 2024-08-06T18:31:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:12 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3103737230' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:13.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:12 smithi107 ceph-mon[32381]: osdmap e734: 12 total, 3 up, 11 in 2024-08-06T18:31:13.469 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:13.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:13.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:13.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:13.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:13.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:14.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:14.032 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:13 smithi117 ceph-mon[32835]: pgmap v2166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:14.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:14.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:13 smithi086 ceph-mon[29229]: pgmap v2166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:14.254 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:14.254 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:14.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:14.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:14.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:13 smithi107 ceph-mon[32381]: pgmap v2166: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:14.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:14.675 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:14.775 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:14.776 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:14.776 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:14.792 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:14.793 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:14.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:14.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:14.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:14.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:14.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:14.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:14.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:14.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:14.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:14.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:14.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:14.861 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:14.861 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:14.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:14.969 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:15.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:15.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4216384532' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:15.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:15.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:15.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:15.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4216384532' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:15.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:15.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:15.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:15.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4216384532' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:15.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:15.473 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:15.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:15.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:15.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:15.700 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:15.701 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:31:15.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:15.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:15.949 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:16.010 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:16.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:31:16.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:16 smithi107 ceph-mon[32381]: pgmap v2167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:16.300 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:16 smithi117 ceph-mon[32835]: pgmap v2167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:16.300 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.301 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:16 smithi086 ceph-mon[29229]: pgmap v2167: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:16.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:16.410 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:16.480 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:16.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:16.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:16.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:16.943 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:16.965 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:16.965 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:17.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:17.189 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:17.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:17.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:17.225 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:17.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:17.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:17.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:17.531 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:17.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:17.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:17.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:17 smithi107 ceph-mon[32381]: pgmap v2168: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:17 smithi107 ceph-mon[32381]: osdmap e735: 12 total, 3 up, 11 in 2024-08-06T18:31:18.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:18.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:17 smithi086 ceph-mon[29229]: pgmap v2168: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:18.104 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:17 smithi086 ceph-mon[29229]: osdmap e735: 12 total, 3 up, 11 in 2024-08-06T18:31:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:17 smithi117 ceph-mon[32835]: pgmap v2168: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:17 smithi117 ceph-mon[32835]: osdmap e735: 12 total, 3 up, 11 in 2024-08-06T18:31:18.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:18.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:18.328 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:18.328 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:18.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:18.481 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:31:18.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:18.714 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:18.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:18.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:18.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:18.736 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:18.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:18.879 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:31:18.879 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:31:18.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:19.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:19.241 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:19.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:19.507 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:19.585 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:31:19.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:19.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:19.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:19.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:20.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:20.054 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:19 smithi107 ceph-mon[32381]: pgmap v2170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:20.054 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:20.054 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:20.080 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:19 smithi086 ceph-mon[29229]: pgmap v2170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:19 smithi117 ceph-mon[32835]: pgmap v2170: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:20.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:20.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:20.431 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:20.431 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:20.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:20.580 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:20.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:20.842 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:20.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:21.107 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:21.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:21.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:21.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:21.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:21.769 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:21.831 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:31:21.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:21.923 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:21 smithi086 ceph-mon[29229]: pgmap v2171: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:21.924 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:21.924 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:21.924 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:22.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:21 smithi117 ceph-mon[32835]: pgmap v2171: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:22.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:22.170 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:21 smithi107 ceph-mon[32381]: pgmap v2171: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:22.307 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:22.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:22.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:22.596 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:22.597 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:22.668 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:22.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:22.841 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:22.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:22 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:22.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:22.848 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:22 smithi086 ceph-mon[29229]: osdmap e736: 12 total, 3 up, 11 in 2024-08-06T18:31:22.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:22.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:22.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:23.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:22 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:22 smithi117 ceph-mon[32835]: osdmap e736: 12 total, 3 up, 11 in 2024-08-06T18:31:23.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:22 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:23.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4144995398' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:23.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:22 smithi107 ceph-mon[32381]: osdmap e736: 12 total, 3 up, 11 in 2024-08-06T18:31:23.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:23.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:23.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:23.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:23.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:23.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:24.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:24.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:23 smithi086 ceph-mon[29229]: pgmap v2173: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:23 smithi117 ceph-mon[32835]: pgmap v2173: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:24.188 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:23 smithi107 ceph-mon[32381]: pgmap v2173: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:24.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:24.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:24.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:24.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:24.388 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:24.388 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:24.389 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:31:24.389 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:31:24.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:24.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:24.685 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:31:24.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:24.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:24.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:25.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:25.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:24 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:31:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:24 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:31:25.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:24 smithi086 ceph-mon[29229]: osdmap e737: 12 total, 3 up, 11 in 2024-08-06T18:31:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:24 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:31:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:24 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:31:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:24 smithi117 ceph-mon[32835]: osdmap e737: 12 total, 3 up, 11 in 2024-08-06T18:31:25.131 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:25.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:24 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:31:25.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:24 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:31:25.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:24 smithi107 ceph-mon[32381]: osdmap e737: 12 total, 3 up, 11 in 2024-08-06T18:31:25.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:25.354 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:31:25.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:25.538 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:25.620 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:25.620 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:25.621 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:25.636 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:25.637 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:25.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:25.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:25.637 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:25.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:25.657 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:25.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:25.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:25.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:25.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:25.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:25.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:25.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:25.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:25.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:25.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:26.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:26.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:25 smithi086 ceph-mon[29229]: pgmap v2175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:26.118 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/13053387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:26.118 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:26.118 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:25 smithi117 ceph-mon[32835]: pgmap v2175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/13053387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:26.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:26.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:26.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:25 smithi107 ceph-mon[32381]: pgmap v2175: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/13053387' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:26.338 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:26.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:26.377 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:26.439 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:26.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:26.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:26.741 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:26.762 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:26.762 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:26.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:26.895 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:26.895 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:26.895 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:26 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:27.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:27.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:26 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:26 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:27.272 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:27.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:27.512 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:27.541 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:27.673 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:27.809 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:27.939 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:28.021 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:27 smithi086 ceph-mon[29229]: pgmap v2176: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:28.075 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:27 smithi117 ceph-mon[32835]: pgmap v2176: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:28.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:27 smithi107 ceph-mon[32381]: pgmap v2176: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:28.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:28.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:28.343 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:28.378 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:31:28.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:28.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:28.748 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:28.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:29.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:29.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:29.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:29.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:673,from:0,reservation_nonce:0)", 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 218.59295056400001, 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017925499999999999, 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:31:29.257 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.258 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430574+0000", 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-07 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430577+0000", 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.864e-06 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:31:29.259 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430587+0000", 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0581000000000001e-05 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430593+0000", 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7060000000000001e-06 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430636+0000", 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2914000000000003e-05 2024-08-06T18:31:29.260 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430649+0000", 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3098999999999999e-05 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430752+0000", 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330799999999999 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:31:29.261 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 218.59230265799999, 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6106999999999996e-05, 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:31:29.262 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431222+0000", 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4899999999999995e-07 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.263 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431225+0000", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2949999999999998e-06 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431233+0000", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8660000000000006e-06 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431237+0000", 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1880000000000004e-06 2024-08-06T18:31:29.264 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431266+0000", 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8912e-05 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431271+0000", 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5330000000000002e-06 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.265 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431287+0000", 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5764e-05 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:697,from:0,reservation_nonce:0)", 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 142.79988137500001, 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000184933, 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:31:29.266 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.267 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223643+0000", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0699999999999996e-07 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223646+0000", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5220000000000002e-06 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223655+0000", 2024-08-06T18:31:29.268 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1509999999999993e-06 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223660+0000", 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3759999999999999e-06 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223696+0000", 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.269 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223709+0000", 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3013999999999999e-05 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223827+0000", 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118313 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.270 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 142.79927337800001, 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7805000000000005e-05, 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.271 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224251+0000", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8699999999999995e-07 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224254+0000", 2024-08-06T18:31:29.272 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7010000000000001e-06 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224261+0000", 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224266+0000", 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3579999999999996e-06 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:31:29.273 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224302+0000", 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6529999999999998e-05 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224310+0000", 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8839999999999994e-06 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224328+0000", 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7901e-05 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.274 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:31:29.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.275 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:31:29.275 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:31:29.275 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:31:29.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:29.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:29.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:29.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:29.708 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:29.708 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:29.817 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:29.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:29.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:29 smithi107 ceph-mon[32381]: pgmap v2177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:29.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:31:30.004 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:30.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:29 smithi117 ceph-mon[32835]: pgmap v2177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:30.128 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:29 smithi086 ceph-mon[29229]: pgmap v2177: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:30.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:30.222 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:30.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:30.382 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:30.382 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:30.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:30.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:30.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:30.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:31.010 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:31.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:31.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:31.404 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:31.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:31.666 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:31.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:31.830 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:31.872 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:31:31.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:31 smithi107 ceph-mon[32381]: pgmap v2178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:31:32.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:31 smithi117 ceph-mon[32835]: pgmap v2178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:31:32.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:32.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:31 smithi086 ceph-mon[29229]: pgmap v2178: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:32.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:31:32.303 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:32.303 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:32.303 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:32.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:32.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:32.462 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:32.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:31:32.513 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:32.513 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:32.514 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:32.532 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:32.532 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:32.532 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:32.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:32.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:32.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:32.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:32.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:32.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:32.727 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:32.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:32.738 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:32.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:32.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:32 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:31:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:32 smithi086 ceph-mon[29229]: osdmap e738: 12 total, 3 up, 11 in 2024-08-06T18:31:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:32 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2287509691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:33.004 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:33.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:32 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:33.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:31:33.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:32 smithi107 ceph-mon[32381]: osdmap e738: 12 total, 3 up, 11 in 2024-08-06T18:31:33.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:32 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2287509691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:33.094 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:33.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:32 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2190303935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:31:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:32 smithi117 ceph-mon[32835]: osdmap e738: 12 total, 3 up, 11 in 2024-08-06T18:31:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:32 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2287509691' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:33.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:33.247 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:33.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:33.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:33.446 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:33.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:33.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:33.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:33.802 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:33.803 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:33.803 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:33.803 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:33.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:33.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:33 smithi107 ceph-mon[32381]: pgmap v2180: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:33.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:33.989 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:34.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:34.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:33 smithi086 ceph-mon[29229]: pgmap v2180: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:34.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:34.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:34.166 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:34.292 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:34.295 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:34.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:34.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:33 smithi117 ceph-mon[32835]: pgmap v2180: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:34.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:34.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:34.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:34.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:34.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:31:34.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:34.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:34.899 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:34.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:31:34.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:35.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:35.099 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:35.150 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:35.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:35.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:35.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:31:35.491 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:35.575 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:35.575 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:35.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:35.757 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:35.844 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:35.845 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:35.892 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:36.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:36.049 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:35 smithi117 ceph-mon[32835]: pgmap v2181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:36.128 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:35 smithi107 ceph-mon[32381]: pgmap v2181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:36.160 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:36.239 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:35 smithi086 ceph-mon[29229]: pgmap v2181: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:36.239 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:36.239 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:36.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:36.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:36.526 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:36.527 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:36.532 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:36.561 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:36.673 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:36.693 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:36.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:36.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:37.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:37.216 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:37.330 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:37.331 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:31:37.352 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:37.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:37.583 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:37.637 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:37.704 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:37.705 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:37.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:37.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:37 smithi086 ceph-mon[29229]: pgmap v2182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:37.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:37 smithi086 ceph-mon[29229]: osdmap e739: 12 total, 3 up, 11 in 2024-08-06T18:31:37.884 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:37.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:37.885 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:37.885 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:31:37.901 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:37.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:37.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:37 smithi107 ceph-mon[32381]: pgmap v2182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:37 smithi107 ceph-mon[32381]: osdmap e739: 12 total, 3 up, 11 in 2024-08-06T18:31:37.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:38.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:37 smithi117 ceph-mon[32835]: pgmap v2182: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:37 smithi117 ceph-mon[32835]: osdmap e739: 12 total, 3 up, 11 in 2024-08-06T18:31:38.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:38.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:38.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:38.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:38.458 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:38.459 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:38.566 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:38.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:38.833 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:38.864 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:38.864 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:38.864 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: osdmap e740: 12 total, 3 up, 11 in 2024-08-06T18:31:38.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4062179716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:38.881 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:38.881 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:38.881 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:38.881 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:38.882 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:38.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:38.899 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:38.901 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:38.902 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:38.902 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:38.902 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:38.966 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: osdmap e740: 12 total, 3 up, 11 in 2024-08-06T18:31:39.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4062179716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:39.096 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:39.102 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:39.102 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2714367302' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: osdmap e740: 12 total, 3 up, 11 in 2024-08-06T18:31:39.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4062179716' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:39.141 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:39.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:39.358 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:39.488 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:39.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:39.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:39.736 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:39.756 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:39.797 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:39.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:31:39.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:39.986 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:39 smithi086 ceph-mon[29229]: pgmap v2185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:39.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:39.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:39.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.016 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:40.087 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:39 smithi117 ceph-mon[32835]: pgmap v2185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.152 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:40.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:39 smithi107 ceph-mon[32381]: pgmap v2185: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:40.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.173 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:40.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:40.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:40.304 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:40.413 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:40.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:40.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:40.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:40.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:40.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:40.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:41.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:41.141 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:41.141 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:41.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:41.341 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:41.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:41.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:41.655 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:41.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:41.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:31:41.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:41.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:41.996 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:41.998 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:42.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:41 smithi086 ceph-mon[29229]: pgmap v2186: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:42.020 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:41 smithi086 ceph-mon[29229]: osdmap e741: 12 total, 3 up, 11 in 2024-08-06T18:31:42.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:31:42.130 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:42.156 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:41 smithi117 ceph-mon[32835]: pgmap v2186: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:42.156 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:41 smithi117 ceph-mon[32835]: osdmap e741: 12 total, 3 up, 11 in 2024-08-06T18:31:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:41 smithi107 ceph-mon[32381]: pgmap v2186: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:41 smithi107 ceph-mon[32381]: osdmap e741: 12 total, 3 up, 11 in 2024-08-06T18:31:42.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:42.267 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:42.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:31:42.282 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:42.282 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:42.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:42.515 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:42.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:42.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:42.768 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:42.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:42.912 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:42.912 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:42.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:43.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:43.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:43.319 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:43.455 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:43.515 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:43.515 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:43.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:43.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:43.853 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:43.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:43.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:43.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:43.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:43.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:44.102 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:44.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:44.148 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:43 smithi117 ceph-mon[32835]: pgmap v2188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:44.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:43 smithi086 ceph-mon[29229]: pgmap v2188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:43 smithi107 ceph-mon[32381]: pgmap v2188: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:44.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:44.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:44.461 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:44.461 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:44.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:44.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:44.772 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:44.772 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:44.772 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:44.772 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:44.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:44.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:31:44.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:45.053 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:45.146 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:45.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:45.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:45.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:45.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:45.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:45.329 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:45.458 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:45.556 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:45.557 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:45.557 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:45.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:45.573 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:45.573 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:45.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:45.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:45.590 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:45.591 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:45.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:45.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:45.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:45.591 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:45.592 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:45.594 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:45.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:45.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:45.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:45.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:45.896 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:31:45.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:46.116 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:46.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:45 smithi086 ceph-mon[29229]: pgmap v2189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:46.178 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:46.178 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3757607311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:46.202 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:45 smithi107 ceph-mon[32381]: pgmap v2189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:46.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:46.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3757607311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:46.252 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:46.278 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:45 smithi117 ceph-mon[32835]: pgmap v2189: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:46.278 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:31:46.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3757607311' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:46.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:46.341 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:46.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:46.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:46.541 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:46.541 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:46.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:46.705 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:31:46.705 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:31:46.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:46.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:47.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:47.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:47.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: osdmap e742: 12 total, 3 up, 11 in 2024-08-06T18:31:47.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:46 smithi117 ceph-mon[32835]: osdmap e743: 12 total, 3 up, 11 in 2024-08-06T18:31:47.182 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: osdmap e742: 12 total, 3 up, 11 in 2024-08-06T18:31:47.203 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:46 smithi107 ceph-mon[32381]: osdmap e743: 12 total, 3 up, 11 in 2024-08-06T18:31:47.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4217461784' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: osdmap e742: 12 total, 3 up, 11 in 2024-08-06T18:31:47.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:46 smithi086 ceph-mon[29229]: osdmap e743: 12 total, 3 up, 11 in 2024-08-06T18:31:47.353 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:31:47.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:47.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:47.606 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:47.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:47.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:47.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:47.846 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:47.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:48.000 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:47 smithi107 ceph-mon[32381]: pgmap v2191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:48.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:48.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:48.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:47 smithi086 ceph-mon[29229]: pgmap v2191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:48.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:48.293 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:48.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:31:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:47 smithi117 ceph-mon[32835]: pgmap v2191: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:48.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:48.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:48.560 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:48.561 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:48.595 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:48.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:48.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:48.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:49.027 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:31:49.067 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:49.180 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:31:49.180 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:31:49.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:49.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:49.337 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:49.337 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:49.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:49.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:49.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:49.746 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:49.746 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:49.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:31:49.869 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:49.916 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:50.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:50.027 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:49 smithi117 ceph-mon[32835]: pgmap v2193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:50.041 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:49 smithi086 ceph-mon[29229]: pgmap v2193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:50.119 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:50.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:49 smithi107 ceph-mon[32381]: pgmap v2193: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:50.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:50.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:50.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:50.638 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:50.638 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:50.638 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:50.644 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:50.644 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:50.645 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:50.645 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:50.645 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:50.662 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:50.662 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:50.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:50.662 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:50.663 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:50.663 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:50.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:50.664 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:50.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:50.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:51.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:51.088 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:50 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3611325385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:51.088 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:51.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:51.223 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:50 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3611325385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:51.223 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:51.332 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:51.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:50 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3611325385' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:51.354 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:51.406 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:51.406 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:51.409 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:51.410 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:51.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:51.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:51.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:51.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:51.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.766 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:599,from:0,reservation_nonce:0)", 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 503.98551502200002, 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.000188605, 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.767 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782370+0000", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782371+0000", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8299999999999995e-07 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782375+0000", 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7780000000000001e-06 2024-08-06T18:31:51.768 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782384+0000", 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.986e-06 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782392+0000", 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.6650000000000003e-06 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.769 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782430+0000", 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.8778e-05 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782444+0000", 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.3981999999999999e-05 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.782559+0000", 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00011443299999999999 2024-08-06T18:31:51.770 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 503.98486674100002, 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.5184000000000005e-05, 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.771 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783018+0000", 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.772 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783019+0000", 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.7199999999999999e-07 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783022+0000", 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.4340000000000001e-06 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783029+0000", 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6590000000000001e-06 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.773 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783032+0000", 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.2650000000000001e-06 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783061+0000", 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.8751999999999998e-05 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.774 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783067+0000", 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7790000000000001e-06 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:23:27.783084+0000", 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.6823000000000001e-05 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.775 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:649,from:0,reservation_nonce:0)", 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 335.668842606, 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021522199999999999, 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.776 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.777 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099044+0000", 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.719e-06 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099049+0000", 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4410000000000003e-06 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099061+0000", 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2422999999999999e-05 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.778 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099070+0000", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6349999999999995e-06 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099113+0000", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3393999999999998e-05 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099128+0000", 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5233e-05 2024-08-06T18:31:51.779 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099258+0000", 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012937699999999999 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 335.66800580500001, 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4050999999999999e-05, 2024-08-06T18:31:51.780 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.781 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099880+0000", 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5199999999999996e-07 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099886+0000", 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1040000000000003e-06 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.782 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099896+0000", 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8789999999999998e-06 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099901+0000", 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970000000000003e-06 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099938+0000", 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6542999999999999e-05 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.783 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099944+0000", 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099963+0000", 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998e-05 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.784 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:699,from:0,reservation_nonce:0)", 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 160.47446704199999, 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026914199999999999, 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.785 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293420+0000", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9920000000000002e-06 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293427+0000", 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6719999999999998e-06 2024-08-06T18:31:51.786 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293442+0000", 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5124e-05 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293453+0000", 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2024-08-06T18:31:51.787 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293534+0000", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0717999999999995e-05 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293552+0000", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7982000000000002e-05 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293687+0000", 2024-08-06T18:31:51.788 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013563200000000001 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 160.473659621, 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9434999999999999e-05, 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:31:51.789 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:31:51.790 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000002e-07 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294229+0000", 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9280000000000002e-06 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294255+0000", 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6409000000000001e-05 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:31:51.791 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294261+0000", 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6300000000000003e-06 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294298+0000", 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7233000000000001e-05 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294304+0000", 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7960000000000001e-06 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:31:51.792 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294325+0000", 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0930000000000001e-05 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:31:51.793 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:31:51.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:51.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:52.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:52.012 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:52.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:51 smithi086 ceph-mon[29229]: pgmap v2194: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:52.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.013 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:51 smithi107 ceph-mon[32381]: pgmap v2194: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:52.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:51 smithi117 ceph-mon[32835]: pgmap v2194: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:52.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:52.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:52.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:52.391 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:52.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:52.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:31:52.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:52.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:52.845 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:52.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:52.951 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:52.951 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:53.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:53.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:53.065 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:53.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:53.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:53.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:31:53.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:53.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:53.662 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:53.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:53.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:53.921 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:31:53.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:54.111 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:53 smithi086 ceph-mon[29229]: pgmap v2195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:54.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:54.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:53 smithi107 ceph-mon[32381]: pgmap v2195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:54.148 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:54.245 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:54.271 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:53 smithi117 ceph-mon[32835]: pgmap v2195: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:54.271 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:31:54.342 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:54.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:54.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:54.642 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:54.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:31:54.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:54.931 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:54.931 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:31:55.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:55.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:55.069 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:55.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:54 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:55.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:55.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:54 smithi086 ceph-mon[29229]: osdmap e744: 12 total, 3 up, 11 in 2024-08-06T18:31:55.180 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:55.180 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:54 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:54 smithi107 ceph-mon[32381]: osdmap e744: 12 total, 3 up, 11 in 2024-08-06T18:31:55.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:55.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:54 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:31:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3527413031' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:31:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:54 smithi117 ceph-mon[32835]: osdmap e744: 12 total, 3 up, 11 in 2024-08-06T18:31:55.444 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:55.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:55.588 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:55.588 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:55.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:31:55.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:55.861 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:55.861 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:55.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:56.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:55 smithi086 ceph-mon[29229]: pgmap v2197: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:56.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:56.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:56.126 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:56.132 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:31:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:31:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:31:56.149 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:31:56.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:31:56.149 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:31:56.149 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:31:56.150 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:31:56.166 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:31:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:31:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:31:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:31:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:31:56.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:31:56.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:31:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:55 smithi107 ceph-mon[32381]: pgmap v2197: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:56.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:55 smithi117 ceph-mon[32835]: pgmap v2197: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:56.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:56.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:56.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:56.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:56.868 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:56.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:31:56.971 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:31:57.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:57.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:31:57.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:57.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:57.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:57.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:57.142 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: osdmap e745: 12 total, 3 up, 11 in 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1441003176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:56 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:57.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:57.345 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:57.345 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:57.345 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: osdmap e745: 12 total, 3 up, 11 in 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1441003176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.346 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:56 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:31:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:31:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: osdmap e745: 12 total, 3 up, 11 in 2024-08-06T18:31:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1441003176' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:31:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:56 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:31:57.379 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:57.445 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:57.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:31:57.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:31:57.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:57.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:57.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:57.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:31:57.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:31:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:31:57.695 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:31:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:57.715 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:57.851 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:57.982 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:58.088 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:57 smithi107 ceph-mon[32381]: pgmap v2199: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:58.114 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:57 smithi117 ceph-mon[32835]: pgmap v2199: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:58.228 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:57 smithi086 ceph-mon[29229]: pgmap v2199: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:31:58.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:31:58.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:31:58.516 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:58.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:31:58.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:31:58.916 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:58.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:31:59.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:59.156 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:31:59.167 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:59.167 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:59.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:31:59.242 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:31:59.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:59.442 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:31:59.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:31:59.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:31:59.463 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:59.463 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:59.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:31:59.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:31:59.725 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:31:59.725 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:31:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:31:59.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:31:59.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:31:59.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:31:59.954 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:31:59.965 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:00.095 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:59 smithi086 ceph-mon[29229]: pgmap v2200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:31:59 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:32:00.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:59 smithi117 ceph-mon[32835]: pgmap v2200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:00.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:00.117 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:31:59 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:32:00.150 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:31:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:59 smithi107 ceph-mon[32381]: pgmap v2200: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:31:59 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:32:00.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:00.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:00.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:00.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:00.751 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:00.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:01.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: osdmap e746: 12 total, 3 up, 11 in 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:32:01.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:32:01.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:01.240 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:01.241 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:01.241 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: osdmap e746: 12 total, 3 up, 11 in 2024-08-06T18:32:01.241 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:32:01.241 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:32:01.241 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:32:01.276 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:01.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:01.297 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3896964069' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: osdmap e746: 12 total, 3 up, 11 in 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:32:01.298 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:32:01.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:01.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:01.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:01.743 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:01.773 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:01.774 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:01.774 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:01.790 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:01.791 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:01.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:01.808 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:01.808 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:01.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:01.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:02.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:01 smithi086 ceph-mon[29229]: pgmap v2202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1645488937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:01 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:02.085 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:01 smithi086 ceph-mon[29229]: osdmap e747: 12 total, 3 up, 11 in 2024-08-06T18:32:02.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:01 smithi107 ceph-mon[32381]: pgmap v2202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1645488937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:01 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:01 smithi107 ceph-mon[32381]: osdmap e747: 12 total, 3 up, 11 in 2024-08-06T18:32:02.245 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:01 smithi117 ceph-mon[32835]: pgmap v2202: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:02.245 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1645488937' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:02.245 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:01 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:02.245 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:01 smithi117 ceph-mon[32835]: osdmap e747: 12 total, 3 up, 11 in 2024-08-06T18:32:02.324 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:02.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:32:02.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:02.555 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:02.555 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:02.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:02.659 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:02.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:32:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:02.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:02.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:02.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:02.912 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:02.986 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:03.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.121 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:03.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:03.269 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.270 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:03.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:03.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:03.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:03.517 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:03.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:03.787 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:03.921 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:04.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:04.094 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:04.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:04.211 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:03 smithi117 ceph-mon[32835]: pgmap v2204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:03 smithi107 ceph-mon[32381]: pgmap v2204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:04.303 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:03 smithi086 ceph-mon[29229]: pgmap v2204: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:04.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:04.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:04.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:32:04.590 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:04.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:32:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:04.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:04.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:04.838 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:04.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:04.914 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:04.975 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:32:04.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:05.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:05.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:05.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:05.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:05.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:05.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:05.350 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:05.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:05.516 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:05.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:05.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:05.922 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:05.972 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:05.972 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:06.056 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:06.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:05 smithi107 ceph-mon[32381]: pgmap v2205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:05 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:06.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:06.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:05 smithi107 ceph-mon[32381]: osdmap e748: 12 total, 3 up, 11 in 2024-08-06T18:32:06.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:05 smithi117 ceph-mon[32835]: pgmap v2205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:06.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:05 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:06.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:06.239 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:05 smithi117 ceph-mon[32835]: osdmap e748: 12 total, 3 up, 11 in 2024-08-06T18:32:06.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:06.351 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:05 smithi086 ceph-mon[29229]: pgmap v2205: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:06.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:05 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:06.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3425862043' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:06.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:05 smithi086 ceph-mon[29229]: osdmap e748: 12 total, 3 up, 11 in 2024-08-06T18:32:06.412 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:06.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:06.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:06.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:06.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:06.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:06.850 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:06.981 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:07.082 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:07.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:32:07.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:07.133 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:07.134 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:07.134 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:07.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:07.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:07.167 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:07.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:07.169 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:07.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:07.329 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:07.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:07.514 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:07.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:07.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:07.650 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:07.781 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:07.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:07.934 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:07.934 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:07.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:07 smithi086 ceph-mon[29229]: pgmap v2207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:07.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:07 smithi086 ceph-mon[29229]: osdmap e749: 12 total, 3 up, 11 in 2024-08-06T18:32:07.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4175897434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:07.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:07.934 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:07.993 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:07.993 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:08.042 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:07 smithi117 ceph-mon[32835]: pgmap v2207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:07 smithi117 ceph-mon[32835]: osdmap e749: 12 total, 3 up, 11 in 2024-08-06T18:32:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4175897434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:08.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:08.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:32:08.208 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:08.208 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:07 smithi107 ceph-mon[32381]: pgmap v2207: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 333 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:07 smithi107 ceph-mon[32381]: osdmap e749: 12 total, 3 up, 11 in 2024-08-06T18:32:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4175897434' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:08.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:08.433 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:08.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:08.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:08.700 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:08.752 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:08.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:08.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:09.104 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:09.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:08 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:09.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:08 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:08 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:09.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:09.362 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:09.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:32:09.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:09.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:09.610 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:09.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:09.764 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:09.899 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:09.954 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:09 smithi086 ceph-mon[29229]: pgmap v2209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:09 smithi107 ceph-mon[32381]: pgmap v2209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:09.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:32:10.023 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:09 smithi117 ceph-mon[32835]: pgmap v2209: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:10.031 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:10.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:10.273 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:10.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:10.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:10.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:10.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:10.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:10.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:10.854 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:32:10.894 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:10.946 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:10.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:11.082 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:11.085 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:11.111 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:11.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:11.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:11.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:11.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:11.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:11.506 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:11.506 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:11.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:11.694 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:11.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:32:11.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:11.767 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:11.767 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:11.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:11.941 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:12.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:12.028 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:12.028 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:12.028 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:12.029 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: pgmap v2210: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: osdmap e750: 12 total, 3 up, 11 in 2024-08-06T18:32:12.029 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:11 smithi086 ceph-mon[29229]: osdmap e751: 12 total, 3 up, 11 in 2024-08-06T18:32:12.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: pgmap v2210: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:12.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:32:12.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:32:12.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:12.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:12.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: osdmap e750: 12 total, 3 up, 11 in 2024-08-06T18:32:12.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:11 smithi117 ceph-mon[32835]: osdmap e751: 12 total, 3 up, 11 in 2024-08-06T18:32:12.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:12.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: pgmap v2210: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2039404336' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: osdmap e750: 12 total, 3 up, 11 in 2024-08-06T18:32:12.163 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:11 smithi107 ceph-mon[32381]: osdmap e751: 12 total, 3 up, 11 in 2024-08-06T18:32:12.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:12.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:12.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:12.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:12.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:12.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:12.558 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:12.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:12.658 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:12.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:12.673 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:12.674 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:12.674 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:12.674 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:12.675 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:12.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:12.691 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:12.692 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:12.692 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:12.692 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:12.692 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:12.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:12.693 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:12.822 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:12.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:13.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:13.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3243007397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:13.116 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3243007397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:13.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3243007397' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:13.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:13.345 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:13.361 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:13.432 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:13.432 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:13.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:13.630 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:13.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:32:13.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:13.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:13 smithi117 ceph-mon[32835]: pgmap v2213: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:13.898 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:13.899 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:13.899 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:13.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:13.900 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:13.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:32:13.910 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:14.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:13 smithi086 ceph-mon[29229]: pgmap v2213: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:14.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.033 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:14.150 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:14.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:13 smithi107 ceph-mon[32381]: pgmap v2213: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:14.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:14.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:14.557 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:14.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:14.580 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:14.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:14.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:14.842 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:14.842 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:14.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:15.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:15.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:15.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:15.433 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:32:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:15.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:15.612 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:15.675 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:15.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:15.760 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:15.883 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:15.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:15.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:15.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:15 smithi107 ceph-mon[32381]: pgmap v2214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:15.985 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:15 smithi117 ceph-mon[32835]: pgmap v2214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:16.013 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:16.112 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:15 smithi086 ceph-mon[29229]: pgmap v2214: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:16.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:16.280 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:16.280 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:32:16.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:16.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:16.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:16.526 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:16.546 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:16.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:16.705 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:16.705 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:16.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:16.862 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:32:16.922 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:16.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:17.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:17.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:17.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:17.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:17.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:17.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:17.339 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:17.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:17.490 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:17.490 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:17.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:17.738 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:17.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:17.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:17 smithi086 ceph-mon[29229]: pgmap v2215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:17.982 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:17 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:17.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:17.983 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:17 smithi086 ceph-mon[29229]: osdmap e752: 12 total, 3 up, 11 in 2024-08-06T18:32:18.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:18.030 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:17 smithi107 ceph-mon[32381]: pgmap v2215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:18.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:17 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:18.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:18.031 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:17 smithi107 ceph-mon[32381]: osdmap e752: 12 total, 3 up, 11 in 2024-08-06T18:32:18.045 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:17 smithi117 ceph-mon[32835]: pgmap v2215: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:17 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4146806252' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:18.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:17 smithi117 ceph-mon[32835]: osdmap e752: 12 total, 3 up, 11 in 2024-08-06T18:32:18.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:18.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:18.402 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:32:18.404+0000 7fb09aaa7640 0 --2- 172.21.15.86:0/682709658 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fb094107c30 0x7fb09410e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:32:18.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:18.425 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:18.425 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:18.425 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:18.442 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:18.443 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:18.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:18.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:18.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:18.461 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:18.461 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:18.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:18.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:18.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:18.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:18.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:18.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:18.575 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:18.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:32:18.680 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:18.701 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:18.701 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:18.810 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:18.820 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:18.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:18.961 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3409216840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:18.962 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:18.962 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:19.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:19.073 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:19.073 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:19.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3409216840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:19.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3409216840' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:19.330 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:19.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:19.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:19.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:19.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:19.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:19.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:19 smithi107 ceph-mon[32381]: pgmap v2217: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:19.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:19.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:19.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:19 smithi117 ceph-mon[32835]: pgmap v2217: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:20.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.005 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:32:20.064 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:20.192 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:20.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:20.275 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:20.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:20.349 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:20.349 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:20.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:19 smithi086 ceph-mon[29229]: pgmap v2217: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:20.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.349 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:20.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:20.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:20.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:20.748 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:20.749 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:32:20.854 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:20.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:21.038 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:21.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:32:21.115 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:21.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:21.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:21.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:21.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:21.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:21.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:21.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:21.673 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:21.673 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:21.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:21.905 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:22.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:21 smithi086 ceph-mon[29229]: pgmap v2218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:22.008 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:21 smithi086 ceph-mon[29229]: osdmap e753: 12 total, 3 up, 11 in 2024-08-06T18:32:22.041 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:22.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:21 smithi117 ceph-mon[32835]: pgmap v2218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:22.063 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:21 smithi117 ceph-mon[32835]: osdmap e753: 12 total, 3 up, 11 in 2024-08-06T18:32:22.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:21 smithi107 ceph-mon[32381]: pgmap v2218: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:21 smithi107 ceph-mon[32381]: osdmap e753: 12 total, 3 up, 11 in 2024-08-06T18:32:22.307 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:22.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:22.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:22.673 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:22.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:22.832 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:22.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:22.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:22.876 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:32:22.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:23.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:23.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:23.126 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:23.237 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:23.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:32:23.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:23.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:23.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:23.484 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:23.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:23.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:23.759 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:23.759 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:23.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:23.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:24.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:24.064 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:23 smithi107 ceph-mon[32381]: pgmap v2220: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:24.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:23 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:24.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:24.065 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:23 smithi107 ceph-mon[32381]: osdmap e754: 12 total, 3 up, 11 in 2024-08-06T18:32:24.078 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:24.078 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:24.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:23 smithi086 ceph-mon[29229]: pgmap v2220: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:24.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:23 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:24.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:24.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:23 smithi086 ceph-mon[29229]: osdmap e754: 12 total, 3 up, 11 in 2024-08-06T18:32:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:23 smithi117 ceph-mon[32835]: pgmap v2220: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:23 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:24.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1669191969' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:24.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:23 smithi117 ceph-mon[32835]: osdmap e754: 12 total, 3 up, 11 in 2024-08-06T18:32:24.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:24.176 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:24.176 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:24.176 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:24.193 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:24.193 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:24.194 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:24.210 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:24.211 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:24.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:24.308 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:24.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:24.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:24.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:24.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:24.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:24.911 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:24.977 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:24.981 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:24.982 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:25.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2017408936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.134 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2017408936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:25.135 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.135 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:25.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2017408936' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:25.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:25.266 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:25.266 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:25.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:25.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:25.600 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:25.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:32:25.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:25.773 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:25.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:32:25.799 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:25.799 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:25.850 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:32:25.909 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:26.036 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:26.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:26.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:25 smithi086 ceph-mon[29229]: pgmap v2222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:26.165 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:26.165 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:25 smithi117 ceph-mon[32835]: pgmap v2222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:26.172 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:25 smithi107 ceph-mon[32381]: pgmap v2222: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:26.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:26.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:26.457 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:26.458 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:26.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:26.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:26.718 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:26.718 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:26.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:26.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:26.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:32:26.985 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:26.985 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:27.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:27.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:27.232 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:27.316 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:27.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:27.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:27.630 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:27.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:27.783 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:27.783 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:27.844 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:27.844 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:27.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:27.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:27 smithi117 ceph-mon[32835]: pgmap v2223: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:27.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:27 smithi117 ceph-mon[32835]: osdmap e755: 12 total, 3 up, 11 in 2024-08-06T18:32:27.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:27 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:32:27.913 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:27 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:32:27.914 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:27.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:32:28.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:28.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:27 smithi086 ceph-mon[29229]: pgmap v2223: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:28.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:27 smithi086 ceph-mon[29229]: osdmap e755: 12 total, 3 up, 11 in 2024-08-06T18:32:28.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:27 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:32:28.079 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:27 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:32:28.156 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:27 smithi107 ceph-mon[32381]: pgmap v2223: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:27 smithi107 ceph-mon[32381]: osdmap e755: 12 total, 3 up, 11 in 2024-08-06T18:32:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:27 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:32:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:27 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:32:28.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:28.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:28.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:28.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:28.554 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:28.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:28.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:28.890 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:32:28.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:28 smithi107 ceph-mon[32381]: osdmap e756: 12 total, 3 up, 11 in 2024-08-06T18:32:28.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:29.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:29.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:28 smithi117 ceph-mon[32835]: osdmap e756: 12 total, 3 up, 11 in 2024-08-06T18:32:29.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:29.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:28 smithi086 ceph-mon[29229]: osdmap e756: 12 total, 3 up, 11 in 2024-08-06T18:32:29.195 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:29.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:29.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:29.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:29.553 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:29.554 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:29.600 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:29.613 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:29.747 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:29.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:30.020 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:29 smithi086 ceph-mon[29229]: pgmap v2226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:29 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:30.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:29 smithi086 ceph-mon[29229]: osdmap e757: 12 total, 3 up, 11 in 2024-08-06T18:32:30.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:30.184 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:29 smithi117 ceph-mon[32835]: pgmap v2226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:30.184 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:29 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:30.184 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:30.184 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:29 smithi117 ceph-mon[32835]: osdmap e757: 12 total, 3 up, 11 in 2024-08-06T18:32:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:29 smithi107 ceph-mon[32381]: pgmap v2226: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:29 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:32:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2161315461' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:29 smithi107 ceph-mon[32381]: osdmap e757: 12 total, 3 up, 11 in 2024-08-06T18:32:30.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:30.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:30.563 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:30.695 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:30.787 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:30.787 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:30.787 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:30.792 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:30.792 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:30.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:30.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:30.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:30.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:30.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:30.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:30.989 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3119390073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:31.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3119390073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3119390073' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:31.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:31.365 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:31.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:31.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:32:31.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:31.764 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:31.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:31.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:31 smithi086 ceph-mon[29229]: pgmap v2228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:31.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:31.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:31.912 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:31.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:31.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:32.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:32.127 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:32.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:32.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:31 smithi107 ceph-mon[32381]: pgmap v2228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:32.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.194 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:32.355 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:31 smithi117 ceph-mon[32835]: pgmap v2228: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:32.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:32.570 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:32.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:32.703 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:32.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:32.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:32.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:33.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:33.203 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:33.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:32:33.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:33.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:33.488 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:33.488 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:33.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:33.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:33.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:33.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:33.964 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:34.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:34.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:33 smithi117 ceph-mon[32835]: pgmap v2229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:34.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:33 smithi107 ceph-mon[32381]: pgmap v2229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:34.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:34.337 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:33 smithi086 ceph-mon[29229]: pgmap v2229: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:34.337 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:34.337 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:34.338 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:34.338 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:34.446 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:34.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:34.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:34.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:34.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:32:34.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:35.125 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:35.149 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:35.149 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:35.149 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:34 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:32:35.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:34 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:32:35.150 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:34 smithi107 ceph-mon[32381]: osdmap e758: 12 total, 3 up, 11 in 2024-08-06T18:32:35.258 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:35.284 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:34 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:32:35.284 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:34 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:32:35.284 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:34 smithi117 ceph-mon[32835]: osdmap e758: 12 total, 3 up, 11 in 2024-08-06T18:32:35.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:34 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:32:35.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:34 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:32:35.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:34 smithi086 ceph-mon[29229]: osdmap e758: 12 total, 3 up, 11 in 2024-08-06T18:32:35.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:35.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:35.639 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:35.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:35.796 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:35.909 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:32:35.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:36.068 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:36.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:35 smithi107 ceph-mon[32381]: pgmap v2231: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:36.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:36.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:35 smithi086 ceph-mon[29229]: pgmap v2231: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:36.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:36.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:36.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:35 smithi117 ceph-mon[32835]: pgmap v2231: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:32:36.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:36.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:36.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:36.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:36.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:36.893 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:37.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:37.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:36 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:37.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:37.031 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:36 smithi086 ceph-mon[29229]: osdmap e759: 12 total, 3 up, 11 in 2024-08-06T18:32:37.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:37.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:37.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:37.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:36 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:37.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:37.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:36 smithi107 ceph-mon[32381]: osdmap e759: 12 total, 3 up, 11 in 2024-08-06T18:32:37.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:36 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2166653836' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:32:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:36 smithi117 ceph-mon[32835]: osdmap e759: 12 total, 3 up, 11 in 2024-08-06T18:32:37.405 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:37.428 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:37.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:37.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:37.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:37.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:37.968 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:37 smithi117 ceph-mon[32835]: pgmap v2233: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:38.078 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:38.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:37 smithi086 ceph-mon[29229]: pgmap v2233: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:37 smithi107 ceph-mon[32381]: pgmap v2233: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:38.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:38.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:38.480 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:38.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:38.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:38.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:38.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:32:38.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:38.902 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:38.902 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:39.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:39.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:39.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:39.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:39.467 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:39.467 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:39.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:39.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:39.784 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:39.804 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:39.804 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:39.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:39.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:39.823 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:39.823 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:39.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:39.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:39.824 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:39.841 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:39.841 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:39.841 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:39.842 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:39.842 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:39.842 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:39.842 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:39.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:40.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:40.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:39 smithi117 ceph-mon[32835]: pgmap v2234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:40.096 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2667243333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:40.097 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:39 smithi086 ceph-mon[29229]: pgmap v2234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2667243333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:40.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:40.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:40.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:39 smithi107 ceph-mon[32381]: pgmap v2234: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2667243333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:40.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:32:40.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:40.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:40.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:40.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:40.737 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:40.875 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:41.010 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:41.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:41.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.171 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:41.171 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:41.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:41.290 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.290 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:41.413 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:41.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:41.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:41.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:41.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:41.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:42.025 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:42.025 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:42.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:42.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:41 smithi117 ceph-mon[32835]: pgmap v2235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:42.193 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:41 smithi086 ceph-mon[29229]: pgmap v2235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:42.196 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:32:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:41 smithi107 ceph-mon[32381]: pgmap v2235: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:42.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:42.359 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:42.495 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:42.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:42.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:42.898 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:32:42.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:32:43.006 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:43.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:43.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:43.182 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:43.182 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:43.182 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:43.182 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:43.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:43.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:43.569 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:43.701 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:43.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:43.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:44.026 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:32:44.104 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:43 smithi107 ceph-mon[32381]: pgmap v2236: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:44.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:44.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:44.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:44.261 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:43 smithi117 ceph-mon[32835]: pgmap v2236: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:44.331 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:44.331 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:44.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:43 smithi086 ceph-mon[29229]: pgmap v2236: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:44.335 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:44.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:44.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:44.635 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:44.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:44.906 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:32:45.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:45.066 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:32:45.067 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:32:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:32:45.067 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:45.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:45.310 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:45.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:45.585 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:45.721 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:45.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:45.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:46.049 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:45 smithi086 ceph-mon[29229]: pgmap v2237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:46.049 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:46.049 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:46.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:46.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:45 smithi107 ceph-mon[32381]: pgmap v2237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:46.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:45 smithi117 ceph-mon[32835]: pgmap v2237: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:46.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:46.532 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:46.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:46.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:46.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:46.803 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:46.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:46.943 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:46.943 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:46.949 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:32:47.072 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:47.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:47.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:47.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:47.233 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:32:47.344 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:47.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:47.607 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:32:47.608 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:32:47.608 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:32:47.608 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:32:47.616 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:47.752 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:47.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:48.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:48.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:47 smithi086 ceph-mon[29229]: pgmap v2238: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:48.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:47 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:48.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:48.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:47 smithi086 ceph-mon[29229]: osdmap e760: 12 total, 3 up, 11 in 2024-08-06T18:32:48.034 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:48.035 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:48.143 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:47 smithi107 ceph-mon[32381]: pgmap v2238: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:47 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:48.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:47 smithi107 ceph-mon[32381]: osdmap e760: 12 total, 3 up, 11 in 2024-08-06T18:32:48.273 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:48.295 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:48.295 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:47 smithi117 ceph-mon[32835]: pgmap v2238: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:47 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:48.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2125204442' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:32:48.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:47 smithi117 ceph-mon[32835]: osdmap e760: 12 total, 3 up, 11 in 2024-08-06T18:32:48.404 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:48.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:32:48.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:48.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:48.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:48.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:48.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:48.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:48.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:48.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:49.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:49.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:49.348 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:49.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:49.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:49.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:49.649 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:49.650 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:49.650 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:49.658 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:49.659 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:49.659 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:49.659 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:49.659 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:49.679 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:49.679 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:49.679 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:49.679 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:49.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:49.680 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:49.681 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:49.686 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:49.750 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:49.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:50.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:50.047 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:49 smithi107 ceph-mon[32381]: pgmap v2240: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:50.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4292151942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:50.048 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:50.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:50.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:50.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:49 smithi086 ceph-mon[29229]: pgmap v2240: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:50.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4292151942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:50.357 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:49 smithi117 ceph-mon[32835]: pgmap v2240: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4292151942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:50.391 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:50.391 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:50.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:50.587 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:50.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:50.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:50.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:51.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:51.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:51.142 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.142 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.269 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:51.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:51.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:32:51.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:51.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:51.650 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:51.694 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:51.824 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:51.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:51.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:51 smithi086 ceph-mon[29229]: pgmap v2241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:51.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:51 smithi086 ceph-mon[29229]: osdmap e761: 12 total, 3 up, 11 in 2024-08-06T18:32:52.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:52.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:51 smithi117 ceph-mon[32835]: pgmap v2241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:52.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:51 smithi117 ceph-mon[32835]: osdmap e761: 12 total, 3 up, 11 in 2024-08-06T18:32:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:51 smithi107 ceph-mon[32381]: pgmap v2241: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:52.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:51 smithi107 ceph-mon[32381]: osdmap e761: 12 total, 3 up, 11 in 2024-08-06T18:32:52.217 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:52.319 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:52.319 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:52.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:52.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:32:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:32:52.419 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:32:52.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:52.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:52.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:52.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:52.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:52.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:53.007 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:53.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:53.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:53.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:53.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:32:53.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:53.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:53.430 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:53.430 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:53.540 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:53.670 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:53.691 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:53.691 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:53.800 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:53.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:53.960 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:53.960 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:54.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:54.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:53 smithi117 ceph-mon[32835]: pgmap v2243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:54.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:53 smithi107 ceph-mon[32381]: pgmap v2243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:54.231 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:53 smithi086 ceph-mon[29229]: pgmap v2243: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:54.340 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:54.475 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:54.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:54.659 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:54.659 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:32:54.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:54.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:54.965 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:32:55.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:55.114 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:55.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:55.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:55.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:32:55.427 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:55.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:55.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:55.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:55.976 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:56.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:55 smithi117 ceph-mon[32835]: pgmap v2244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:55 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:56.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:55 smithi117 ceph-mon[32835]: osdmap e762: 12 total, 3 up, 11 in 2024-08-06T18:32:56.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:56.272 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:55 smithi107 ceph-mon[32381]: pgmap v2244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:56.272 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:55 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:56.272 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:56.273 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:55 smithi107 ceph-mon[32381]: osdmap e762: 12 total, 3 up, 11 in 2024-08-06T18:32:56.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:55 smithi086 ceph-mon[29229]: pgmap v2244: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:56.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:55 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:32:56.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1066821556' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:32:56.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:55 smithi086 ceph-mon[29229]: osdmap e762: 12 total, 3 up, 11 in 2024-08-06T18:32:56.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:56.521 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:32:56.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:56.675 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:32:56.675 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:32:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:32:56.784 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:56.919 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:57.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:57.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:57.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:32:57.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:57.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:32:57.724 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:57.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:57 smithi117 ceph-mon[32835]: pgmap v2246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:57 smithi117 ceph-mon[32835]: osdmap e763: 12 total, 3 up, 11 in 2024-08-06T18:32:57.873 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/81305386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:57.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:58.091 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:32:58.091 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:32:58.091 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:32:58.091 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:32:58.092 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:32:58.093 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:32:58.093 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:32:58.093 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:32:58.093 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:32:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:57 smithi086 ceph-mon[29229]: pgmap v2246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:57 smithi086 ceph-mon[29229]: osdmap e763: 12 total, 3 up, 11 in 2024-08-06T18:32:58.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/81305386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:58.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:58.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:57 smithi107 ceph-mon[32381]: pgmap v2246: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:32:58.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:57 smithi107 ceph-mon[32381]: osdmap e763: 12 total, 3 up, 11 in 2024-08-06T18:32:58.151 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/81305386' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:32:58.259 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:58.390 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:32:58.526 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:58.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:58.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:32:58.852 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:32:58.853 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:32:58.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:58.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:58.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:58.958 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.071 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:32:59.076 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:32:59.095 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:32:59.096 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:32:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:32:59.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:32:59.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:599,from:6,reservation_nonce:26)", 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:32:59.330 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 571.54883852399996, 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00018547099999999999, 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:32:59.331 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782024+0000", 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782025+0000", 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5340000000000001e-06 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.332 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782035+0000", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0999999999999993e-06 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782055+0000", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0650999999999999e-05 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782064+0000", 2024-08-06T18:32:59.333 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9209999999999998e-06 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782156+0000", 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1822000000000003e-05 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782172+0000", 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5611999999999999e-05 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.334 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782209+0000", 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7830999999999997e-05 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:599/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:32:59.335 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 571.54819717199996, 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111396, 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.336 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782665+0000", 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782666+0000", 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.6899999999999998e-07 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782670+0000", 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.048e-06 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.337 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782697+0000", 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6948999999999999e-05 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782702+0000", 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8199999999999996e-06 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.338 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782739+0000", 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7166999999999998e-05 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782745+0000", 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.1430000000000003e-06 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.782777+0000", 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.18e-05 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.339 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:599,from:6,reservation_nonce:0)", 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 571.54750413800002, 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136124, 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.340 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783358+0000", 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783359+0000", 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1699999999999999e-07 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.341 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783361+0000", 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5359999999999999e-06 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783367+0000", 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.285e-06 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.342 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783376+0000", 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4780000000000008e-06 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783463+0000", 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.6978999999999995e-05 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783480+0000", 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.6634e-05 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.343 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:23:27.783494+0000", 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4895000000000001e-05 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:649,from:6,reservation_nonce:28)", 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 403.232095716, 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147861, 2024-08-06T18:32:59.344 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.345 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098768+0000", 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4e-07 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.346 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098773+0000", 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9139999999999999e-06 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098783+0000", 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0508e-05 2024-08-06T18:32:59.347 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098790+0000", 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5409999999999997e-06 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098871+0000", 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1824999999999997e-05 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.348 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098887+0000", 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5617e-05 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098915+0000", 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7515999999999999e-05 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:649/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:32:59.349 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 403.23134805500001, 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013723899999999999, 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.350 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099515+0000", 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6300000000000005e-07 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099522+0000", 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0110000000000003e-06 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.351 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099533+0000", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1272e-05 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099540+0000", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4670000000000001e-06 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099619+0000", 2024-08-06T18:32:59.352 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9609000000000003e-05 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099628+0000", 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5110000000000003e-06 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099652+0000", 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3706e-05 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.353 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:673,from:2,reservation_nonce:27)", 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.90049999399997, 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145129, 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.354 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430364+0000", 2024-08-06T18:32:59.355 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.513e-06 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430371+0000", 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0190000000000001e-06 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430385+0000", 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.356 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430396+0000", 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0229e-05 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430466+0000", 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0858000000000005e-05 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430481+0000", 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4596999999999999e-05 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.357 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430508+0000", 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:673/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.89989225099998, 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111407, 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.358 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.359 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430971+0000", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6599999999999996e-07 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430975+0000", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2889999999999999e-06 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430986+0000", 2024-08-06T18:32:59.360 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0729e-05 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430992+0000", 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.694e-06 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431040+0000", 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8418999999999998e-05 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.361 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431050+0000", 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431082+0000", 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1754999999999997e-05 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.362 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:673,from:2,reservation_nonce:0)", 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 309.89927902300002, 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012740599999999999, 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.363 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431584+0000", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.36e-07 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431586+0000", 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0080000000000001e-06 2024-08-06T18:32:59.364 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431590+0000", 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0090000000000001e-06 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431596+0000", 2024-08-06T18:32:59.365 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2759999999999997e-06 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431681+0000", 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4974000000000003e-05 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431696+0000", 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4654000000000001e-05 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.366 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431711+0000", 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5248999999999999e-05 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:697,from:2,reservation_nonce:28)", 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 234.10741169299999, 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014356300000000001, 2024-08-06T18:32:59.367 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.368 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223452+0000", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3510000000000001e-06 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223460+0000", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9049999999999997e-06 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223475+0000", 2024-08-06T18:32:59.369 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5021999999999999e-05 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223483+0000", 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0350000000000001e-06 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223540+0000", 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7235999999999999e-05 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.370 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223561+0000", 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0309000000000002e-05 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223594+0000", 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3704999999999997e-05 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.371 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:697,from:2,reservation_nonce:0)", 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 234.10623831800001, 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110475, 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.372 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.77e-07 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224627+0000", 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.424e-06 2024-08-06T18:32:59.373 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224633+0000", 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2169999999999999e-06 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224638+0000", 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0010000000000001e-06 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.374 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224712+0000", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4124999999999999e-05 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224723+0000", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0895e-05 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224735+0000", 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1436e-05 2024-08-06T18:32:59.375 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:699,from:6,reservation_nonce:30)", 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 228.03778415599999, 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022859099999999999, 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.376 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293081+0000", 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4099999999999998e-06 2024-08-06T18:32:59.377 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293094+0000", 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2809999999999999e-05 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293125+0000", 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0998999999999999e-05 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.378 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293139+0000", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4667e-05 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293256+0000", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116816 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293272+0000", 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5574e-05 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.379 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293307+0000", 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5315000000000003e-05 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:699,from:6,reservation_nonce:0)", 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 228.03617550199999, 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016018, 2024-08-06T18:32:59.380 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.381 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7e-07 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294707+0000", 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9466e-05 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:32:59.382 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294716+0000", 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6660000000000003e-06 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294722+0000", 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6489999999999996e-06 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294825+0000", 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102699 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.383 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294838+0000", 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3339999999999999e-05 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294847+0000", 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9900000000000003e-06 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:32:59.384 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:32:59.385 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:32:59.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:32:59.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:32:59.656 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:32:59.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:32:59.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:00.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:00.081 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:32:59 smithi086 ceph-mon[29229]: pgmap v2248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:00.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:32:59 smithi117 ceph-mon[32835]: pgmap v2248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:00.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:00.211 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:32:59 smithi107 ceph-mon[32381]: pgmap v2248: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:00.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:33:00.327 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:00.336 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:00.337 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:00.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:00.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:00.721 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:00.742 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:00.742 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:00.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:00.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:33:00.983 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:00.984 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:01.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:33:01.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:33:01.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:33:01.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:00 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:33:01.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:33:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:33:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:33:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:00 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:33:01.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:01.142 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:01.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:33:01.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:33:01.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:33:01.143 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:00 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:33:01.252 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:01.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:01.274 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:01.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:01.513 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:01.646 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:01.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:01.906 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:01.966 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:01 smithi086 ceph-mon[29229]: pgmap v2249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:02.036 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:02.057 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:01 smithi117 ceph-mon[32835]: pgmap v2249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:02.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:02.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:01 smithi107 ceph-mon[32381]: pgmap v2249: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:02.302 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:02.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:02.457 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:02.457 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:02.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:02.696 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:02.828 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:02.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:03.089 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:03.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:03.356 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:03.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:03.514 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:03.514 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:03.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:03.649 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:03.649 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:03.659 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:03.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:03.870 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:33:03.879 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:03.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:03.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:03.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:03.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:03 smithi107 ceph-mon[32381]: pgmap v2250: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:03.920 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:03 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:04.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:04.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:03 smithi117 ceph-mon[32835]: pgmap v2250: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:04.055 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:03 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:04.165 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:04.190 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:04.191 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:04.300 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:04.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:03 smithi086 ceph-mon[29229]: pgmap v2250: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:04.323 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:03 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:04.382 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:04.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:33:04.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:04.568 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:04.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:04.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:04.703 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:04.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:04.968 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:05.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:04 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:05.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:05.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:04 smithi086 ceph-mon[29229]: osdmap e764: 12 total, 3 up, 11 in 2024-08-06T18:33:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:04 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:04 smithi117 ceph-mon[32835]: osdmap e764: 12 total, 3 up, 11 in 2024-08-06T18:33:05.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:04 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:05.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/199466947' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:05.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:04 smithi107 ceph-mon[32381]: osdmap e764: 12 total, 3 up, 11 in 2024-08-06T18:33:05.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:05.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:05.398 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:05.398 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:05.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:05.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:05.779 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:05.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:05.934 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:05 smithi107 ceph-mon[32381]: pgmap v2252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:06.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:06.065 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:05 smithi086 ceph-mon[29229]: pgmap v2252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:06.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:06.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:06.311 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:06.337 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:05 smithi117 ceph-mon[32835]: pgmap v2252: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:06.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:06.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:06.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:06.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:06.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:06.768 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:06.768 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:06.769 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:06.785 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:06.785 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:06.785 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:06.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:06.786 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:06.803 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:06.803 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:06.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:06.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:06.987 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:07.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:06 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:33:07.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:06 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:33:07.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2907211110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:07.019 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:06 smithi086 ceph-mon[29229]: osdmap e765: 12 total, 3 up, 11 in 2024-08-06T18:33:07.123 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:06 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:33:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:06 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:33:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2907211110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:07.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:06 smithi117 ceph-mon[32835]: osdmap e765: 12 total, 3 up, 11 in 2024-08-06T18:33:07.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:06 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:33:07.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:06 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:33:07.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2907211110' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:07.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:06 smithi107 ceph-mon[32381]: osdmap e765: 12 total, 3 up, 11 in 2024-08-06T18:33:07.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:07.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:07.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:07.663 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:07.799 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:07.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:33:07.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:07.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:07 smithi107 ceph-mon[32381]: pgmap v2253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:07.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:07.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:07.960 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.052 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:07 smithi086 ceph-mon[29229]: pgmap v2253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:08.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.053 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.069 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:08.087 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:08.087 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:07 smithi117 ceph-mon[32835]: pgmap v2253: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:08.200 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:08.336 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:08.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:08.600 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:08.677 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:08.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:08.865 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:09.001 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:09.136 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:09.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:09.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:09.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:09.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:09.537 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:09.672 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:09.803 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:09.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:09.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:09.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:10.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:10.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:33:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:09 smithi117 ceph-mon[32835]: pgmap v2255: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:10.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:09 smithi086 ceph-mon[29229]: pgmap v2255: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:10.168 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:10.168 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:10.206 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:09 smithi107 ceph-mon[32381]: pgmap v2255: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:10.337 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:10.357 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:10.469 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:10.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:10.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:10.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:10.741 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:10.876 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:10.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:11.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:11.062 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:11.062 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:11.090 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:11.091 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:33:11.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:11.277 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:11.414 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:11.549 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:11.572 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:11.575 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:11.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:11.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:11.841 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:11.841 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:11.887 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:33:11.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:12.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:12.170 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:11 smithi086 ceph-mon[29229]: pgmap v2256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:12.170 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:11 smithi107 ceph-mon[32381]: pgmap v2256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:12.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:12.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:11 smithi117 ceph-mon[32835]: pgmap v2256: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:12.483 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:12.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:12.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:12.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:12.768 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:12.878 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:13.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:13.036 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:13.037 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:13.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:12 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:13.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:13.037 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:12 smithi086 ceph-mon[29229]: osdmap e766: 12 total, 3 up, 11 in 2024-08-06T18:33:13.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:13.172 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:12 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:13.172 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:13.172 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:12 smithi117 ceph-mon[32835]: osdmap e766: 12 total, 3 up, 11 in 2024-08-06T18:33:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:12 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/409353529' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:12 smithi107 ceph-mon[32381]: osdmap e766: 12 total, 3 up, 11 in 2024-08-06T18:33:13.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:13.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:13.548 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:13.570 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:13.571 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:13.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:13.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:13.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:13.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:13.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:14.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:14.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:13 smithi107 ceph-mon[32381]: pgmap v2258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:14.102 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:14.102 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:14.174 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:13 smithi086 ceph-mon[29229]: pgmap v2258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:14.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:14.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:14.366 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:14.366 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:14.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:13 smithi117 ceph-mon[32835]: pgmap v2258: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:14.475 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:14.606 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:14.689 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:14.689 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:14.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:14.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:15.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:15.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:15.147 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:15.147 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:15.148 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:15.163 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:15.164 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:15.181 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:15.182 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:15.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:15.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:15.182 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:15.183 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:15.183 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:15.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:15.341 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:15.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:15.554 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:15.657 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:33:15.689 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:15.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:15.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:15.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:15 smithi086 ceph-mon[29229]: pgmap v2259: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:15.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:15 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2407407333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:15.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:15.980 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:16.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:33:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:15 smithi117 ceph-mon[32835]: pgmap v2259: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:15 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2407407333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:16.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:15 smithi107 ceph-mon[32381]: pgmap v2259: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:16.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:15 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2407407333' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:16.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.214 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:16.339 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:16.340 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:16.362 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:16.497 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:16.629 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:16.656 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:16.657 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:16.681 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:16.709 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:16.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:16.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:16 smithi107 ceph-mon[32381]: osdmap e767: 12 total, 3 up, 11 in 2024-08-06T18:33:16.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:16.981 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:16 smithi086 ceph-mon[29229]: osdmap e767: 12 total, 3 up, 11 in 2024-08-06T18:33:17.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:17.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:17.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:16 smithi117 ceph-mon[32835]: osdmap e767: 12 total, 3 up, 11 in 2024-08-06T18:33:17.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:17.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:17.324 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:17.324 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:17.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:17.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:17.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:17.840 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:17.935 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:17.971 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:18.107 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:17 smithi086 ceph-mon[29229]: pgmap v2260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:17 smithi107 ceph-mon[32381]: pgmap v2260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:18.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:18.368 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:17 smithi117 ceph-mon[32835]: pgmap v2260: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:18.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:18.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:18.644 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:18.644 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:18.682 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:33:18.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:18.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:18.914 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:33:19.033 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:19.168 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:19.194 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:18 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:33:19.194 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:18 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:33:19.195 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:18 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:33:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:18 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:33:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:19.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:19.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:19.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:19.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:18 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:33:19.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:18 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:33:19.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:19.368 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:19.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:19.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:19.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:19.832 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:19.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:20.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:19 smithi086 ceph-mon[29229]: pgmap v2262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:19 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:20.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:19 smithi086 ceph-mon[29229]: osdmap e768: 12 total, 3 up, 11 in 2024-08-06T18:33:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:19 smithi117 ceph-mon[32835]: pgmap v2262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:19 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:19 smithi117 ceph-mon[32835]: osdmap e768: 12 total, 3 up, 11 in 2024-08-06T18:33:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:19 smithi107 ceph-mon[32381]: pgmap v2262: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:19 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/249882656' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:19 smithi107 ceph-mon[32381]: osdmap e768: 12 total, 3 up, 11 in 2024-08-06T18:33:20.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:20.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:20.501 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:20.631 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:20.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:20.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:21.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:21.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:21.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:21.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:21.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:21.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:21.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:21.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:21.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:21.427 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:21.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:21.584 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:33:21.584 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:33:21.584 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:33:21.584 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:33:21.699 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:21.830 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:21.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:33:21.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:21.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:21.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:22.098 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:21 smithi117 ceph-mon[32835]: pgmap v2264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:21 smithi117 ceph-mon[32835]: osdmap e769: 12 total, 3 up, 11 in 2024-08-06T18:33:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:21 smithi107 ceph-mon[32381]: pgmap v2264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:21 smithi107 ceph-mon[32381]: osdmap e769: 12 total, 3 up, 11 in 2024-08-06T18:33:22.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:22.250 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:22.250 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:21 smithi086 ceph-mon[29229]: pgmap v2264: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:22.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:21 smithi086 ceph-mon[29229]: osdmap e769: 12 total, 3 up, 11 in 2024-08-06T18:33:22.359 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:22.490 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:22.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:22.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:22.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:22.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:22.782 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:33:22.897 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:23.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:23.165 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:23.266 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:23.266 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:23.266 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:23.288 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:23.289 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:23.289 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:23.289 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:23.289 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:23.294 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:23.297 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:23.310 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:23.311 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:23.311 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:23.311 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:23.312 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:23.313 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:23.313 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:23.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:23.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:23.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:23.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:23.972 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:23.973 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:23.974 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:24.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:24.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:23 smithi086 ceph-mon[29229]: pgmap v2266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:24.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/902753376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:24.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.196 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:23 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:23 smithi107 ceph-mon[32381]: pgmap v2266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/902753376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:23 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:23 smithi117 ceph-mon[32835]: pgmap v2266: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/902753376' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:23 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:24.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:24.511 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:24.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:24.622 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:24.645 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:24.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:24.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:25.007 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:25.007 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:25.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:25.180 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:25.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:25.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:25.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:25.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:25.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:25.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:25.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:25.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:25.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:25.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:33:25.977 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:26.111 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:26.135 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:25 smithi107 ceph-mon[32381]: pgmap v2267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:26.181 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:25 smithi086 ceph-mon[29229]: pgmap v2267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:26.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:25 smithi117 ceph-mon[32835]: pgmap v2267: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:26.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:26.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:26.529 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:26.529 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:26.638 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:26.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:26.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:26.867 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:26.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:26.931 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:33:27.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:27.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:27.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:27.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:27.205 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:27.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:27.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:27.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:27.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:27.505 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:27.505 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:27.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:27.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:27.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:27.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:27.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:27.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:28.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:27 smithi086 ceph-mon[29229]: pgmap v2268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:28.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:27 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:28.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:28.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:27 smithi086 ceph-mon[29229]: osdmap e770: 12 total, 3 up, 11 in 2024-08-06T18:33:28.119 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:27 smithi117 ceph-mon[32835]: pgmap v2268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:27 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:27 smithi117 ceph-mon[32835]: osdmap e770: 12 total, 3 up, 11 in 2024-08-06T18:33:28.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:28.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:27 smithi107 ceph-mon[32381]: pgmap v2268: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:27 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2859678701' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:28.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:27 smithi107 ceph-mon[32381]: osdmap e770: 12 total, 3 up, 11 in 2024-08-06T18:33:28.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:28.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:28.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:28.639 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:33:28.639 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:33:28.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:28.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:28.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:29.061 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:29.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:29.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:29.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:29.370 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:33:29.467 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:29.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:33:29.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:29.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:29.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:30.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:30.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:30.158 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:29 smithi107 ceph-mon[32381]: pgmap v2270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:30.186 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:29 smithi086 ceph-mon[29229]: pgmap v2270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:30.186 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:30.186 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:30.267 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:30.293 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:29 smithi117 ceph-mon[32835]: pgmap v2270: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:30.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:30.429 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:30.430 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:30.430 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:30.446 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:30.447 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:30.469 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:30.470 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:30.471 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:30.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:30.670 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:30.671 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:30.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:30.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:30.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3985032016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:30.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:31.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3985032016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:31.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:31.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3985032016' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:31.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:31.209 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:31.239 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:31.240 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:31.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:31.479 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:31.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:31.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:31.885 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:31.919 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:32.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:32.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:31 smithi107 ceph-mon[32381]: pgmap v2271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:32.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:31 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:31 smithi107 ceph-mon[32381]: osdmap e771: 12 total, 3 up, 11 in 2024-08-06T18:33:32.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:32.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:31 smithi117 ceph-mon[32835]: pgmap v2271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:32.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:31 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:31 smithi117 ceph-mon[32835]: osdmap e771: 12 total, 3 up, 11 in 2024-08-06T18:33:32.287 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:32.311 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:31 smithi086 ceph-mon[29229]: pgmap v2271: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:32.311 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.311 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:31 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:32.311 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:31 smithi086 ceph-mon[29229]: osdmap e771: 12 total, 3 up, 11 in 2024-08-06T18:33:32.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:32.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:32.574 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:32.574 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:32.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:32.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:32.982 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:33.105 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:33.105 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:33.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:33.241 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:33:33.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:33.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:33.409 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:33.409 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:33 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:33.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:33.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:33.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:33.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:33.942 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:33:34.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:33 smithi086 ceph-mon[29229]: pgmap v2273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:34.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:33 smithi107 ceph-mon[32381]: pgmap v2273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:34.216 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:33 smithi117 ceph-mon[32835]: pgmap v2273: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:34.216 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:34.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:34.460 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:34.516 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:34.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:34.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:34.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:34.882 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:34.882 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:34.991 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:35.062 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:673,from:0,reservation_nonce:0)", 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 345.63121527999999, 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017925499999999999, 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.063 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:33:35.064 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430574+0000", 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-07 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430577+0000", 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.864e-06 2024-08-06T18:33:35.065 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430587+0000", 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0581000000000001e-05 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430593+0000", 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7060000000000001e-06 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.066 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430636+0000", 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2914000000000003e-05 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430649+0000", 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3098999999999999e-05 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.067 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430752+0000", 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330799999999999 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 345.63056737400001, 2024-08-06T18:33:35.068 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6106999999999996e-05, 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.069 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431222+0000", 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4899999999999995e-07 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.070 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431225+0000", 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2949999999999998e-06 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431233+0000", 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8660000000000006e-06 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.071 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431237+0000", 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1880000000000004e-06 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431266+0000", 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8912e-05 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.072 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431271+0000", 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5330000000000002e-06 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431287+0000", 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5764e-05 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.073 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:697,from:0,reservation_nonce:0)", 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 269.838146091, 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000184933, 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.074 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223643+0000", 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0699999999999996e-07 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223646+0000", 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5220000000000002e-06 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223655+0000", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1509999999999993e-06 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223660+0000", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3759999999999999e-06 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223696+0000", 2024-08-06T18:33:35.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223709+0000", 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3013999999999999e-05 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223827+0000", 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118313 2024-08-06T18:33:35.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 269.83753809400002, 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7805000000000005e-05, 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.079 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224251+0000", 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8699999999999995e-07 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224254+0000", 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7010000000000001e-06 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224261+0000", 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224266+0000", 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3579999999999996e-06 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224302+0000", 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6529999999999998e-05 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224310+0000", 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8839999999999994e-06 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224328+0000", 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7901e-05 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:764,from:0,reservation_nonce:0)", 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 29.221718764999999, 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016942300000000001, 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:33:35.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1999999999999998e-07 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840074+0000", 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1690000000000001e-06 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840082+0000", 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4290000000000001e-06 2024-08-06T18:33:35.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840088+0000", 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7710000000000003e-06 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840124+0000", 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840136+0000", 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2211e-05 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840239+0000", 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103173 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.089 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 29.221098943000001, 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102233, 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:33:35.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:33:35.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840690+0000", 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9699999999999995e-07 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840695+0000", 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2869999999999998e-06 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840703+0000", 2024-08-06T18:33:35.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1419999999999993e-06 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840708+0000", 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1189999999999996e-06 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840751+0000", 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3139999999999997e-05 2024-08-06T18:33:35.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840775+0000", 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3730000000000001e-05 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840792+0000", 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7017999999999999e-05 2024-08-06T18:33:35.094 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:33:35.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:33:35.095 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:33:35.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:35.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:34 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:35.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:35.169 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:34 smithi086 ceph-mon[29229]: osdmap e772: 12 total, 3 up, 11 in 2024-08-06T18:33:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:34 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:34 smithi107 ceph-mon[32381]: osdmap e772: 12 total, 3 up, 11 in 2024-08-06T18:33:35.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:34 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:33:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3148608886' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:34 smithi117 ceph-mon[32835]: osdmap e772: 12 total, 3 up, 11 in 2024-08-06T18:33:35.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:35.517 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:35.518 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:35.518 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:35.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:35.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:35.779 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:35.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:35.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:36.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:35 smithi086 ceph-mon[29229]: pgmap v2275: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:36.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:36.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:35 smithi117 ceph-mon[32835]: pgmap v2275: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:36.187 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:35 smithi107 ceph-mon[32381]: pgmap v2275: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:36.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:36.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:36.470 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:33:36.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:36.725 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:36.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:36.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:36.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:37.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:37.263 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:37.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:37.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:37.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:37.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:37.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:33:37.891 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:37.892 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:37.892 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:37 smithi086 ceph-mon[29229]: pgmap v2276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:37.892 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:37 smithi086 ceph-mon[29229]: osdmap e773: 12 total, 3 up, 11 in 2024-08-06T18:33:37.892 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:37.908 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:37.909 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:37.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:37.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:37.909 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:37.926 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:37.926 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:37.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:37.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:37.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:37.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:37.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:37.928 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:38.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:38.082 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:37 smithi117 ceph-mon[32835]: pgmap v2276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:38.083 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:37 smithi117 ceph-mon[32835]: osdmap e773: 12 total, 3 up, 11 in 2024-08-06T18:33:38.161 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:38.162 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:38.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:37 smithi107 ceph-mon[32381]: pgmap v2276: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:38.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:37 smithi107 ceph-mon[32381]: osdmap e773: 12 total, 3 up, 11 in 2024-08-06T18:33:38.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:38.321 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:38.366 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:38.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:38.584 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:38.699 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:38.699 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:38.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:38.728 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:38.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:38.855 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:38.856 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:38.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:38.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4164157865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:38.994 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:38.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:38.995 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:38.995 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:38.995 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:39.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:39.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4164157865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:39.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.118 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.119 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:39.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:33:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4164157865' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:39.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:39.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:39.436 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:39.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:39.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:39.645 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:39.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:39.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:39.987 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:39 smithi086 ceph-mon[29229]: pgmap v2278: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:40.039 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:40.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:39 smithi117 ceph-mon[32835]: pgmap v2278: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:40.174 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:39 smithi107 ceph-mon[32381]: pgmap v2278: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:40.305 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:40.437 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:40.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:40.499 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:40.569 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:40.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:33:40.703 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:40.875 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:40.882 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:33:41.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:41.044 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:41.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:41.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:41.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:41.141 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:41.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:41.272 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:41.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:33:41.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:41.424 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:41.424 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:41.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:41.592 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:41.593 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:33:41.663 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:41.674 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:41.794 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:41.838 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:41.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:42.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:42.031 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:42.032 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:42.032 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:42.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:41 smithi086 ceph-mon[29229]: pgmap v2279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:42.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:41 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:42.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:42.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:41 smithi086 ceph-mon[29229]: osdmap e774: 12 total, 3 up, 11 in 2024-08-06T18:33:42.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:41 smithi086 ceph-mon[29229]: osdmap e775: 12 total, 3 up, 11 in 2024-08-06T18:33:42.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:42.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:42.126 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:42.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:41 smithi107 ceph-mon[32381]: pgmap v2279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:42.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:41 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:42.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:42.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:41 smithi107 ceph-mon[32381]: osdmap e774: 12 total, 3 up, 11 in 2024-08-06T18:33:42.127 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:41 smithi107 ceph-mon[32381]: osdmap e775: 12 total, 3 up, 11 in 2024-08-06T18:33:42.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:42.318 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:42.340 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:42.340 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:41 smithi117 ceph-mon[32835]: pgmap v2279: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:41 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3151974378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:33:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:41 smithi117 ceph-mon[32835]: osdmap e774: 12 total, 3 up, 11 in 2024-08-06T18:33:42.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:41 smithi117 ceph-mon[32835]: osdmap e775: 12 total, 3 up, 11 in 2024-08-06T18:33:42.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:42.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:42.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:42.606 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:42.715 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:42.845 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:42.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:42.979 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:43.109 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:43.199 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:43.199 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:43.240 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:43.375 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:43.421 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:43.422 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:43.423 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:43.423 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:43.437 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:43.437 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:43.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:43.437 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:43.438 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:43.454 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:43.454 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:43.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:43.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:43.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:43.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:43.455 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:43.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:43.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:43.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:33:43.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:43.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:43.797 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:43.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:43 smithi107 ceph-mon[32381]: pgmap v2282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:43 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:33:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:43 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:33:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3511839733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:43.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:44.044 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:44.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:43 smithi086 ceph-mon[29229]: pgmap v2282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:44.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:43 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:33:44.066 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:43 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:33:44.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3511839733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:44.067 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:44.089 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:44.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:33:44.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:44.221 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:44.221 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:44.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:44.334 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:44.342 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:43 smithi117 ceph-mon[32835]: pgmap v2282: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:43 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:33:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:43 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:33:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3511839733' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:44.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:44.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:44.632 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:44.632 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:44.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:44.730 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:44.730 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:44.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:44.996 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:45.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.022 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:44 smithi117 ceph-mon[32835]: osdmap e776: 12 total, 3 up, 11 in 2024-08-06T18:33:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:44 smithi086 ceph-mon[29229]: osdmap e776: 12 total, 3 up, 11 in 2024-08-06T18:33:45.132 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:45.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:44 smithi107 ceph-mon[32381]: osdmap e776: 12 total, 3 up, 11 in 2024-08-06T18:33:45.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:45.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:45.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:45.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:45.543 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:45.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:45.789 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:45.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:45.812 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:45.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:45.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:45.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:45.930 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:46.060 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:45 smithi086 ceph-mon[29229]: pgmap v2284: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:46.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:46.192 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:46.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:45 smithi107 ceph-mon[32381]: pgmap v2284: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:46.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:33:46.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:45 smithi117 ceph-mon[32835]: pgmap v2284: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:46.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:33:46.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:46.469 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:46.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:46.702 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:46.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:46.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:46.865 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:33:46.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:46 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:47.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:47.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:46 smithi086 ceph-mon[29229]: osdmap e777: 12 total, 3 up, 11 in 2024-08-06T18:33:47.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:47.144 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:47.144 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:46 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:47.144 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:47.144 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:46 smithi107 ceph-mon[32381]: osdmap e777: 12 total, 3 up, 11 in 2024-08-06T18:33:47.255 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:46 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1884767454' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:33:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:46 smithi117 ceph-mon[32835]: osdmap e777: 12 total, 3 up, 11 in 2024-08-06T18:33:47.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:47.504 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:47.504 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:47.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:47.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:47.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:47.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:48.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:48.066 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:47 smithi107 ceph-mon[32381]: pgmap v2285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:48.066 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:48.066 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:48.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:48.209 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:47 smithi086 ceph-mon[29229]: pgmap v2285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:48.270 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:48.317 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:47 smithi117 ceph-mon[32835]: pgmap v2285: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:48.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:48.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:48.583 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:48.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:48.834 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:48.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:33:48.849 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:48.927 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:48.927 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:48.927 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:48.944 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:48.945 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:48.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:48.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:48.945 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:48.961 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:48.962 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:48.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:48.962 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:48.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:48.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:48.963 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:49.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:49.086 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:49.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:49.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:49.403 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:49.459 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:49.459 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:49.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:49.666 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:49.736 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:49.737 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:49.802 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:49.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:49.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:49.935 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:49.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:49.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:49 smithi107 ceph-mon[32381]: pgmap v2287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1819990441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:49.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:49 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:50.025 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:49 smithi117 ceph-mon[32835]: pgmap v2287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:50.025 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1819990441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:50.025 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:50.026 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:49 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:50.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:50.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:50.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:49 smithi086 ceph-mon[29229]: pgmap v2287: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 334 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:50.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1819990441' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:50.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:50.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:49 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:50.329 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:50.352 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:50.352 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:50.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:33:50.461 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:50.598 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:50.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:50.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:50.697 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:50.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:50.733 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:33:50.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:50.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:50.907 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:50.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:51.070 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:51.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:51.121 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:51.151 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:51.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:51.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:51.389 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:33:51.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:33:51.513 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:51.638 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:51.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:51.738 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:33:51.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:51.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:51.798 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:51.905 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:51.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:51.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:51.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:51 smithi107 ceph-mon[32381]: pgmap v2288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:52.038 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:52.059 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:51 smithi086 ceph-mon[29229]: pgmap v2288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:52.168 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:52.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:51 smithi117 ceph-mon[32835]: pgmap v2288: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:52.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:52.442 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:52.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:52.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:52.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:52.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:52.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:52.912 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:33:52.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:52.977 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:53.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:53.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:53.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:53.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:33:53.340 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:33:53.340 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:33:53.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:53.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:53.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:53.517 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:53.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:53.758 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:53.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:53.782 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:53.890 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:54.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:33:54.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:54.080 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:54.080 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:53 smithi086 ceph-mon[29229]: pgmap v2289: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:53 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:54.080 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:53 smithi086 ceph-mon[29229]: osdmap e778: 12 total, 3 up, 11 in 2024-08-06T18:33:54.162 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:54.186 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:53 smithi117 ceph-mon[32835]: pgmap v2289: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:54.187 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:53 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:54.187 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:54.187 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:53 smithi117 ceph-mon[32835]: osdmap e778: 12 total, 3 up, 11 in 2024-08-06T18:33:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:53 smithi107 ceph-mon[32381]: pgmap v2289: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:53 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:33:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/447192385' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:33:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:53 smithi107 ceph-mon[32381]: osdmap e778: 12 total, 3 up, 11 in 2024-08-06T18:33:54.258 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:54.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:54.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:54.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:54.632 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:54.633 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:54.633 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:54.633 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:54.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:54.633 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:54.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:54.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:54.634 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:54.634 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:54.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:54.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:54.650 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:54.651 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:54.651 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:54.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:54.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:54.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:54.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:54.691 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:54.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:54.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:54.988 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:54 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:33:54.988 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:54 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:33:54.988 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/421655889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:55.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:55.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:54 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:33:55.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:54 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:33:55.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/421655889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:55.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:55.252 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:55.252 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:55.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:54 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:33:55.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:54 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:33:55.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/421655889' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:33:55.361 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:55.384 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:33:55.384 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:33:55.490 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:55.621 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:55.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:33:55.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:55.793 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:33:55.793 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:33:55.793 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:33:55.794 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:33:55.794 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:33:55.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:56.007 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:56.024 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:55 smithi086 ceph-mon[29229]: pgmap v2291: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.046 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:55 smithi107 ceph-mon[32381]: pgmap v2291: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:33:56.298 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:56.298 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:55 smithi117 ceph-mon[32835]: pgmap v2291: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:33:56.415 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:56.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:33:56.551 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:33:56.686 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:56.718 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:56.820 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:33:56.953 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:56.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:57.085 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:57.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:57.240 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:57.240 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:57.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:57.385 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:33:57.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:57.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:57.732 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:57.732 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:57.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:57.814 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:57.815 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:57.874 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:33:57.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:57.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:57 smithi086 ceph-mon[29229]: pgmap v2292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:57.996 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:57 smithi086 ceph-mon[29229]: osdmap e779: 12 total, 3 up, 11 in 2024-08-06T18:33:58.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:33:58.098 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:57 smithi107 ceph-mon[32381]: pgmap v2292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:58.098 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:57 smithi107 ceph-mon[32381]: osdmap e779: 12 total, 3 up, 11 in 2024-08-06T18:33:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:57 smithi117 ceph-mon[32835]: pgmap v2292: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:33:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:57 smithi117 ceph-mon[32835]: osdmap e779: 12 total, 3 up, 11 in 2024-08-06T18:33:58.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:33:58.272 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:58.318 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:33:58.318 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:33:58.318 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:649,from:0,reservation_nonce:0)", 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 462.21974908099997, 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021522199999999999, 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:33:58.319 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:33:58.320 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099044+0000", 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.719e-06 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099049+0000", 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4410000000000003e-06 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099061+0000", 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2422999999999999e-05 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.321 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099070+0000", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6349999999999995e-06 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099113+0000", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3393999999999998e-05 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099128+0000", 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5233e-05 2024-08-06T18:33:58.322 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099258+0000", 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012937699999999999 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 462.21891227999998, 2024-08-06T18:33:58.323 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4050999999999999e-05, 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.324 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099880+0000", 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5199999999999996e-07 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.325 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099886+0000", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1040000000000003e-06 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099896+0000", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8789999999999998e-06 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099901+0000", 2024-08-06T18:33:58.326 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970000000000003e-06 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099938+0000", 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6542999999999999e-05 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:33:58.327 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099944+0000", 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099963+0000", 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998e-05 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:699,from:0,reservation_nonce:0)", 2024-08-06T18:33:58.328 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.02537351699999, 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026914199999999999, 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.329 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293420+0000", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9920000000000002e-06 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293427+0000", 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6719999999999998e-06 2024-08-06T18:33:58.330 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293442+0000", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5124e-05 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293453+0000", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293534+0000", 2024-08-06T18:33:58.331 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0717999999999995e-05 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293552+0000", 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7982000000000002e-05 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293687+0000", 2024-08-06T18:33:58.332 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013563200000000001 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 287.024566096, 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9434999999999999e-05, 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:33:58.333 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.334 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000002e-07 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294229+0000", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9280000000000002e-06 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294255+0000", 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6409000000000001e-05 2024-08-06T18:33:58.335 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294261+0000", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6300000000000003e-06 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294298+0000", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7233000000000001e-05 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294304+0000", 2024-08-06T18:33:58.336 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7960000000000001e-06 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294325+0000", 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0930000000000001e-05 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:33:58.337 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:33:58.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:33:58.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:33:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:33:58.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:33:58.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:58.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:58.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:58.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:58.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:58.795 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:58.876 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:33:58.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:59.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:33:59.058 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:33:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:59.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:59.175 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:33:59.175 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:33:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:33:59.175 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:33:59.175 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:33:59.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:33:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:33:59.279 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:33:59.326 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:59.457 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:33:59.592 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:33:59.651 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:33:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:33:59.652 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:33:59.666 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:33:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:33:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:33:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:33:59.667 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:33:59.684 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:33:59.686 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:33:59.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:33:59.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:33:59.987 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:00.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:59 smithi117 ceph-mon[32835]: pgmap v2294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:59 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:59 smithi117 ceph-mon[32835]: osdmap e780: 12 total, 3 up, 11 in 2024-08-06T18:34:00.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:33:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3784365131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:00.145 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:00.145 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:33:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:00.145 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:59 smithi086 ceph-mon[29229]: pgmap v2294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:00.145 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:59 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:00.145 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:00.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:59 smithi086 ceph-mon[29229]: osdmap e780: 12 total, 3 up, 11 in 2024-08-06T18:34:00.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:33:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3784365131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:59 smithi107 ceph-mon[32381]: pgmap v2294: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:59 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3166228759' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:59 smithi107 ceph-mon[32381]: osdmap e780: 12 total, 3 up, 11 in 2024-08-06T18:34:00.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:33:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3784365131' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:00.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:00.384 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:00.521 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:00.656 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:00.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:00.836 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:00.836 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:00.841 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:00.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:01.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:01.075 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.075 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.076 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.139 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:01.184 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:01.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:01.363 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:01.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:34:01.443 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:01.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:01.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:01.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:01.620 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:34:01.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:01.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:01.854 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:01.854 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:01.973 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:02.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:34:02.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: pgmap v2296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:34:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:01 smithi117 ceph-mon[32835]: osdmap e781: 12 total, 3 up, 11 in 2024-08-06T18:34:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: pgmap v2296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:34:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:34:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:34:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:34:02.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:01 smithi107 ceph-mon[32381]: osdmap e781: 12 total, 3 up, 11 in 2024-08-06T18:34:02.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:02.345 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:02.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:02.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:02.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: pgmap v2296: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:02.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:34:02.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:34:02.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:34:02.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:34:02.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:01 smithi086 ceph-mon[29229]: osdmap e781: 12 total, 3 up, 11 in 2024-08-06T18:34:02.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:02.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:02.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:02.650 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:02.650 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:02.760 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:02.838 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:34:02.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:03.030 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:03.053 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:03.053 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:03.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:03.182 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:03.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:03.422 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:03.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:03.573 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:03.573 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:03.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:03.761 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:34:03.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:03.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:03.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:03.866 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:03.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:04.015 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:34:04.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:04.124 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:04.137 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:03 smithi107 ceph-mon[32381]: pgmap v2298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:04.212 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:04.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:03 smithi086 ceph-mon[29229]: pgmap v2298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:04.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:03 smithi117 ceph-mon[32835]: pgmap v2298: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:04.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:04.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:04.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:04.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:04.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:04.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:04.890 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:34:05.014 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:05.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:04 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:34:05.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:04 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:34:05.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:04 smithi107 ceph-mon[32381]: osdmap e782: 12 total, 3 up, 11 in 2024-08-06T18:34:05.036 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:05.145 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:05.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:04 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:34:05.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:04 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:34:05.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:04 smithi086 ceph-mon[29229]: osdmap e782: 12 total, 3 up, 11 in 2024-08-06T18:34:05.168 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:05.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:05.302 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:04 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:34:05.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:04 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:34:05.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:04 smithi117 ceph-mon[32835]: osdmap e782: 12 total, 3 up, 11 in 2024-08-06T18:34:05.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:05.332 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:05.333 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:05.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:05.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:05.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:05.779 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:05.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:05.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:05.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:05.957 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:06.032 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:06.032 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:06.032 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:05 smithi086 ceph-mon[29229]: pgmap v2300: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:06.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:05 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:06.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:06.033 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:05 smithi086 ceph-mon[29229]: osdmap e783: 12 total, 3 up, 11 in 2024-08-06T18:34:06.089 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:05 smithi117 ceph-mon[32835]: pgmap v2300: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:05 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:06.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:05 smithi117 ceph-mon[32835]: osdmap e783: 12 total, 3 up, 11 in 2024-08-06T18:34:06.140 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:06.141 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:06.148 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:06.149 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:06.166 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:06.167 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:06.168 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:05 smithi107 ceph-mon[32381]: pgmap v2300: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:05 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4226581400' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:05 smithi107 ceph-mon[32381]: osdmap e783: 12 total, 3 up, 11 in 2024-08-06T18:34:06.222 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:06.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:06.392 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:06.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:34:06.493 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:06.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:06.687 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:06.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:06.776 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:06.777 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:06.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:06.896 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/877592118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:06.896 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:06.896 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:06 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:07.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/877592118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:07.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:06 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.149 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:07.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/877592118' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:07.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:06 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.284 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:07.348 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:07.348 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:07.378 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:07.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:07.442 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:07.442 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:07.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:07.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:07.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:07.919 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.920 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:07 smithi117 ceph-mon[32835]: pgmap v2302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:07.952 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:07.974 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:07.974 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:07 smithi086 ceph-mon[29229]: pgmap v2302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:07.975 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:07.975 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:08.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:08.112 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:07 smithi107 ceph-mon[32381]: pgmap v2302: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:08.214 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:08.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:08.482 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:08.613 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:08.666 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:08.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:34:08.743 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:08.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:08.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:08.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:08.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:34:08.915 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:09.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:09.038 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:09.038 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:09.130 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:09.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:09.281 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:09.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:34:09.414 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:09.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:09.509 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:09.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:09.684 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:09.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:09.861 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:09.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:09.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:09.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:09.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: pgmap v2303: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=4067689800,config=507150784,health=1838778340,kv=880662715,logm=561639927}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526772948}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666037509}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873787329}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485634687}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044788703}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856331}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191038354}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3737072602}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4231049538}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072137503}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951889076}) 2024-08-06T18:34:09.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440536858}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156135628}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089259420}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632215294}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4015108172}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330530654}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944719508}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296402218}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428290725}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493987381}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057890954}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198924925}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875900697}) 2024-08-06T18:34:09.964 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512843272}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911562687}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704677595}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411536337}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756553765}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603479068}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047253761}) 2024-08-06T18:34:09.965 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792455322}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2881198319}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058761766}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112955719}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3636202460}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477317377}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938305856}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137618072}) 2024-08-06T18:34:09.966 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761174815}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415223986}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2486884684}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765744070}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484890930}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792794448}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317394043}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606106169}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069141892}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180023875}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2907967720}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502950148}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3297841921}) 2024-08-06T18:34:09.967 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100856345}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272864746}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265551188}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417667405}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180934299}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767417388}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239454476}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566424699}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225077979}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092473487}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817573433}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131120704}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714722142}) 2024-08-06T18:34:09.968 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629530575}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193523951}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596893686}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037361243}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461935401}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883148664}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992957032}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330629344}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308630777}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367373870}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097012813}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=151023299}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=3726782689,mdsmap=817224545,mgr=2403162910}) 2024-08-06T18:34:09.969 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=64} crc {mgr=2427855960,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3473644415,mon_config_key=3927015083}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3536323564}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1822466236}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=3245957095,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1759850517,osdmap=2979960215}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2103135837}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1459577897}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3338259786}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1597309236}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=899888615}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1880395364}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3305616452}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2031936086}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2381275415}) 2024-08-06T18:34:09.970 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:09 smithi107 ceph-mon[32381]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=324254875}) 2024-08-06T18:34:10.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:10.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: pgmap v2303: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:10.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=4067689800,config=507150784,health=1838778340,kv=880662715,logm=561639927}) 2024-08-06T18:34:10.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526772948}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666037509}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873787329}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485634687}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044788703}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856331}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191038354}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3737072602}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4231049538}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072137503}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951889076}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440536858}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156135628}) 2024-08-06T18:34:10.106 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089259420}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632215294}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4015108172}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330530654}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944719508}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296402218}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428290725}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493987381}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057890954}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198924925}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875900697}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512843272}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911562687}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704677595}) 2024-08-06T18:34:10.107 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411536337}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756553765}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603479068}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047253761}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792455322}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2881198319}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058761766}) 2024-08-06T18:34:10.108 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112955719}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3636202460}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477317377}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938305856}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137618072}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761174815}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415223986}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2486884684}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765744070}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484890930}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792794448}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317394043}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606106169}) 2024-08-06T18:34:10.109 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069141892}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180023875}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2907967720}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502950148}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3297841921}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100856345}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272864746}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265551188}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417667405}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180934299}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767417388}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239454476}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566424699}) 2024-08-06T18:34:10.110 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225077979}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092473487}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817573433}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131120704}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714722142}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629530575}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193523951}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596893686}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037361243}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461935401}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883148664}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992957032}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330629344}) 2024-08-06T18:34:10.111 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308630777}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367373870}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097012813}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=151023299}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=3726782689,mdsmap=817224545,mgr=2403162910}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=64} crc {mgr=2427855960,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3473644415,mon_config_key=3927015083}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3536323564}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1822466236}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=3245957095,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1759850517,osdmap=2979960215}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2103135837}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1459577897}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3338259786}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1597309236}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=899888615}) 2024-08-06T18:34:10.112 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1880395364}) 2024-08-06T18:34:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3305616452}) 2024-08-06T18:34:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:34:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2031936086}) 2024-08-06T18:34:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2381275415}) 2024-08-06T18:34:10.113 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:09 smithi117 ceph-mon[32835]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=324254875}) 2024-08-06T18:34:10.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:10.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: pgmap v2303: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:10.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {auth=29,config=2,health=12,kv=53,logm=4} crc {auth=4067689800,config=507150784,health=1838778340,kv=880662715,logm=561639927}) 2024-08-06T18:34:10.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1526772948}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1666037509}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1873787329}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2485634687}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3044788703}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=856331}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1191038354}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3737072602}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4231049538}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1072137503}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2951889076}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2440536858}) 2024-08-06T18:34:10.247 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1156135628}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2089259420}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=632215294}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4015108172}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330530654}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2944719508}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1296402218}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=428290725}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=493987381}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1057890954}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1198924925}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=875900697}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=512843272}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=911562687}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1704677595}) 2024-08-06T18:34:10.248 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1411536337}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3756553765}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2603479068}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3047253761}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2792455322}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2881198319}) 2024-08-06T18:34:10.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2058761766}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4112955719}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3636202460}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1477317377}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1938305856}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4137618072}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1761174815}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1415223986}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2486884684}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=765744070}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1484890930}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3792794448}) 2024-08-06T18:34:10.250 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3317394043}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2606106169}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4069141892}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180023875}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2907967720}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3502950148}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3297841921}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2100856345}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2272864746}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3265551188}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3417667405}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3180934299}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1767417388}) 2024-08-06T18:34:10.251 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1239454476}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3566424699}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3225077979}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1092473487}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1817573433}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=131120704}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2714722142}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=629530575}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=4193523951}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2596893686}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2037361243}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1461935401}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1883148664}) 2024-08-06T18:34:10.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=992957032}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2330629344}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=1308630777}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=2367373870}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=3097012813}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=100} crc {logm=151023299}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {logm=95,mdsmap=3,mgr=2} crc {logm=3726782689,mdsmap=817224545,mgr=2403162910}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mgr=16,mgr_command_descs=1,mgr_metadata=2,mgrstat=17,mon_config_key=64} crc {mgr=2427855960,mgr_command_descs=1266370550,mgr_metadata=2565076038,mgrstat=3473644415,mon_config_key=3927015083}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=3536323564}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=100} crc {mon_config_key=1822466236}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {mon_config_key=15,monmap=5,osd_metadata=12,osd_pg_creating=1,osdmap=67} crc {mon_config_key=3245957095,monmap=2720365863,osd_metadata=3203987697,osd_pg_creating=1759850517,osdmap=2979960215}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2103135837}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1459577897}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3338259786}) 2024-08-06T18:34:10.253 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1597309236}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=899888615}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=1880395364}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=3305616452}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=4243786056}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2031936086}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=100} crc {osdmap=2381275415}) 2024-08-06T18:34:10.254 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:09 smithi086 ceph-mon[29229]: scrub ok on 0,1,2: ScrubResult(keys {osdmap=73} crc {osdmap=324254875}) 2024-08-06T18:34:10.299 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:10.355 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:10.487 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:10.510 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:10.510 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:10.619 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:10.749 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:10.829 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:10.829 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:10.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:10.938 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:10.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:34:11.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:11.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:11.189 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:11.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:11.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:11.286 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:11.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:11.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:11.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:11.553 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:11.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:11.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:11.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:11.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:11.903 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:34:11.954 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:12.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:12.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:11 smithi086 ceph-mon[29229]: pgmap v2304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:12.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:11 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:11 smithi107 ceph-mon[32381]: pgmap v2304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:11 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:12.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:11 smithi117 ceph-mon[32835]: pgmap v2304: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:12.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:11 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:12.218 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:12.351 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:12.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:12.615 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:12.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:12.654 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:12.673 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:12.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:12.779 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:12.780 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:12.787 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:12.787 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:12.787 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:12.787 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:12.788 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:12.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:13.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:13.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:13.165 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:12 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:13.165 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:13.165 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:12 smithi086 ceph-mon[29229]: osdmap e784: 12 total, 3 up, 11 in 2024-08-06T18:34:13.165 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3066433317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:12 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:12 smithi107 ceph-mon[32381]: osdmap e784: 12 total, 3 up, 11 in 2024-08-06T18:34:13.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3066433317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:13.231 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:12 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:13.232 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2978731493' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:13.232 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:12 smithi117 ceph-mon[32835]: osdmap e784: 12 total, 3 up, 11 in 2024-08-06T18:34:13.232 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3066433317' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:13.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:34:13.274 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:13.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:13.430 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:13.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:13.502 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:13.502 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:13.509 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:13.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:13.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:13.716 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:13.717 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:13.825 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:13.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:14.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:14.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:13 smithi086 ceph-mon[29229]: pgmap v2306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:14.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:13 smithi107 ceph-mon[32381]: pgmap v2306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.217 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:14.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:13 smithi117 ceph-mon[32835]: pgmap v2306: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:14.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.238 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:14.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:14.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:14.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:14.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:14.639 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:14.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:14.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:14.967 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:15.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:15.150 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:15.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:15.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:15.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:15.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:15.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:15.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:34:15.522 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:15.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:34:15.556 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:15.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:15.764 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:15.767 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:15.820 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:15.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:15.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:15.956 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:15.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:34:16.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:16.115 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:15 smithi107 ceph-mon[32381]: pgmap v2307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:16.115 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:16.115 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:16.115 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:16.115 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:16.136 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:15 smithi086 ceph-mon[29229]: pgmap v2307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:16.155 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:15 smithi117 ceph-mon[32835]: pgmap v2307: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:16.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:16.355 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:16.489 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:16.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:16.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:16.620 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:16.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:16.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:16.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:16.883 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:17.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:17.077 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:17.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:17.239 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:17.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:17.420 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:17.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:17.648 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:17.649 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:17.649 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:17.657 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:17.658 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:17.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:17.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:17.658 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:17.675 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:17.675 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:17.676 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:17.676 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:17.676 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:17.676 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:17.676 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:17.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:17.817 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:17.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:34:17.820 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:17.949 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:17 smithi107 ceph-mon[32381]: pgmap v2308: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:17 smithi107 ceph-mon[32381]: osdmap e785: 12 total, 3 up, 11 in 2024-08-06T18:34:17.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4176978559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:18.066 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:18.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:18.100 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:17 smithi117 ceph-mon[32835]: pgmap v2308: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:18.100 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:17 smithi117 ceph-mon[32835]: osdmap e785: 12 total, 3 up, 11 in 2024-08-06T18:34:18.100 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4176978559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:18.160 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:17 smithi086 ceph-mon[29229]: pgmap v2308: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:18.160 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:17 smithi086 ceph-mon[29229]: osdmap e785: 12 total, 3 up, 11 in 2024-08-06T18:34:18.160 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4176978559' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:18.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:18.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:18.471 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:18.522 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:18.522 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:18.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:18.738 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:18.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:19.003 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:19.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.048 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.125 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:19.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:19.285 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:19.286 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:19.286 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.286 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.286 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:19.310 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:19.310 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:19.342 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:19.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:19.539 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:19.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:19.801 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:19.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:34:19.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:19.954 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:19 smithi107 ceph-mon[32381]: pgmap v2310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:20.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:19 smithi086 ceph-mon[29229]: pgmap v2310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:20.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:20.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:20.160 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:20.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:20.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:20.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:20.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:19 smithi117 ceph-mon[32835]: pgmap v2310: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:20.323 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:20.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:20.474 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:20.474 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:20.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:20.712 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:20.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:20.887 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:34:20.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:21.109 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:21.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:21.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:21.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:21.130 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:21.211 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:21.239 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:21.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:34:21.381 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:21.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:21.511 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:21.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:21.517 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:21.580 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:21.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:21.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:21.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:21.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:21 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:21 smithi107 ceph-mon[32381]: pgmap v2311: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:21 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:21 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:21 smithi107 ceph-mon[32381]: osdmap e786: 12 total, 3 up, 11 in 2024-08-06T18:34:21.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:21 smithi107 ceph-mon[32381]: osdmap e787: 12 total, 3 up, 11 in 2024-08-06T18:34:22.041 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:22.090 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:22.090 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:22.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:21 smithi086 ceph-mon[29229]: pgmap v2311: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:22.090 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:21 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:22.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:21 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:22.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:21 smithi086 ceph-mon[29229]: osdmap e786: 12 total, 3 up, 11 in 2024-08-06T18:34:22.091 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:21 smithi086 ceph-mon[29229]: osdmap e787: 12 total, 3 up, 11 in 2024-08-06T18:34:22.118 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:22.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:34:22.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:21 smithi117 ceph-mon[32835]: pgmap v2311: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:22.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:21 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:22.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:21 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3192294681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:22.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:21 smithi117 ceph-mon[32835]: osdmap e786: 12 total, 3 up, 11 in 2024-08-06T18:34:22.140 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:21 smithi117 ceph-mon[32835]: osdmap e787: 12 total, 3 up, 11 in 2024-08-06T18:34:22.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:22.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:22.368 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:22.440 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:22.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:22.705 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:22.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:22.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:23.055 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:23.055 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:23.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:23.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:23.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:23.431 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:23.431 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:23.431 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:23.449 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:23.450 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:23.450 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:23.450 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:23.450 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:23.467 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:23.467 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:23.467 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:23.467 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:23.468 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:23.468 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:23.468 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:23.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:23.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:34:23.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:23.568 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:23.672 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:23.786 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:23.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:23.942 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:23 smithi107 ceph-mon[32381]: pgmap v2314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:23.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2682410819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:24.071 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:23 smithi117 ceph-mon[32835]: pgmap v2314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:24.071 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2682410819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:24.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:24.094 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:24.134 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:24.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:24.188 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:24.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:23 smithi086 ceph-mon[29229]: pgmap v2314: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:24.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2682410819' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:24.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:24.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:24.374 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:24.374 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:34:24.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:24.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:24.623 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:24.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:24.872 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:24.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:24.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:24.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:24.894 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:24.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:24.999 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.000 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.005 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:25.138 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:25.161 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:25.161 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:25.161 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.161 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.162 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:25.270 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:25.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:25.438 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:25.438 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:25.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:25.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:25.819 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:25.956 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:25.978 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:25 smithi086 ceph-mon[29229]: pgmap v2315: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:25.979 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:25.979 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:26.058 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:26.088 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:26.095 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:34:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:25 smithi117 ceph-mon[32835]: pgmap v2315: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:25 smithi107 ceph-mon[32381]: pgmap v2315: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:26.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:26.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:26.229 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:26.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:26.411 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:26.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:26.603 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:26.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:34:26.620 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:26.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:26.797 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:26.797 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:26.848 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:26.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:26.906 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:34:27.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:27.098 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:26 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:27.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:27.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:26 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:27.246 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:26 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:27.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:27.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:27.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:27.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:27.813 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:27.842 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:27.945 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:28.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:28.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:27 smithi117 ceph-mon[32835]: pgmap v2316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:28.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:27 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:28.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:28.101 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:27 smithi117 ceph-mon[32835]: osdmap e788: 12 total, 3 up, 11 in 2024-08-06T18:34:28.102 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:28.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:27 smithi086 ceph-mon[29229]: pgmap v2316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:28.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:27 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:28.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:28.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:27 smithi086 ceph-mon[29229]: osdmap e788: 12 total, 3 up, 11 in 2024-08-06T18:34:28.210 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:27 smithi107 ceph-mon[32381]: pgmap v2316: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:27 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:34:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2952000494' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:27 smithi107 ceph-mon[32381]: osdmap e788: 12 total, 3 up, 11 in 2024-08-06T18:34:28.311 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:28.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:28.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:28.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:28.379 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:28.380 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:28.396 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:28.397 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:28.414 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:28.414 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:28.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:28.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:28.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:28.415 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:28.416 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:28.478 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:28.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:28.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:28.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:34:28.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:29.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:29.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/171146555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:29.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:29.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:29.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:29.123 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:29.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:29.158 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/171146555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:29.159 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:29.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/171146555' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:29.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:29.267 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:29.402 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:29.475 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:29.476 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:29.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:29.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:29.801 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:29.932 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:30.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:30.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.104 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:29 smithi117 ceph-mon[32835]: pgmap v2318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:29 smithi086 ceph-mon[29229]: pgmap v2318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:30.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:30.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:29 smithi107 ceph-mon[32381]: pgmap v2318: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:30.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:30.469 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:30.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:30.493 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:30.555 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:30.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:30.735 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:30.796 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:34:30.822 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:30.822 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:30.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:30.995 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:31.059 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:31.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:31.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:31.127 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:31.162 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:31.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:34:31.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:31.393 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:31.406 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:34:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:31.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:31.477 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:34:31.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:31.483 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:31.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:31.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:31.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:31.806 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:31.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:31.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:31.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:31.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:31 smithi107 ceph-mon[32381]: pgmap v2319: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:31.947 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:31 smithi107 ceph-mon[32381]: osdmap e789: 12 total, 3 up, 11 in 2024-08-06T18:34:31.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:31.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:31.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:31.948 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:32.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:32.099 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:31 smithi117 ceph-mon[32835]: pgmap v2319: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:32.099 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:31 smithi117 ceph-mon[32835]: osdmap e789: 12 total, 3 up, 11 in 2024-08-06T18:34:32.166 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:31 smithi086 ceph-mon[29229]: pgmap v2319: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:32.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:31 smithi086 ceph-mon[29229]: osdmap e789: 12 total, 3 up, 11 in 2024-08-06T18:34:32.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:32.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:32.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:32.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:32.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:32.581 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:32.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:32.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:32.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:32.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:32.963 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:32.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:33.130 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:33.142 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:33.142 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:33.266 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:33.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:33.533 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:33.552 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:33.552 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:34:33.664 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:33.798 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:33.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:33.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:33.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:33.827 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:33.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:34.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:34.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:33 smithi086 ceph-mon[29229]: pgmap v2321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:34.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:34.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:33 smithi107 ceph-mon[32381]: pgmap v2321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:34.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:34.348 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:33 smithi117 ceph-mon[32835]: pgmap v2321: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:34.349 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:34.349 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:34.349 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:34.364 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:34.373 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:34.373 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:34.392 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:34.393 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:34.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:34.393 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:34.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:34.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:34.394 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:34.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:34.592 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:34.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:34.705 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:34.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:34.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:34.928 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:34:34.990 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:35.011 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:35.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3294567696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:35.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:34:35.012 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:34 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:34:35.120 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:35.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:35.147 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3294567696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:34:35.147 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:34 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:34:35.149 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:35.149 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:35.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:34:35.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3294567696' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:34:35.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:34 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:34:35.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:35.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:35.413 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:35.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:35.659 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:35.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:35.920 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:36.055 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:36.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:35 smithi107 ceph-mon[32381]: pgmap v2322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:35 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:36.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:36.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:35 smithi107 ceph-mon[32381]: osdmap e790: 12 total, 3 up, 11 in 2024-08-06T18:34:36.215 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:36.215 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:35 smithi117 ceph-mon[32835]: pgmap v2322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:36.215 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:35 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:36.215 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:36.216 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:35 smithi117 ceph-mon[32835]: osdmap e790: 12 total, 3 up, 11 in 2024-08-06T18:34:36.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:36.237 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:35 smithi086 ceph-mon[29229]: pgmap v2322: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:36.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:35 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:36.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2957448497' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:34:36.238 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:35 smithi086 ceph-mon[29229]: osdmap e790: 12 total, 3 up, 11 in 2024-08-06T18:34:36.325 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:36.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:36.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:36.725 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:36.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:36.877 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:36.878 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:36.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:36.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:36.988 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:37.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:37.150 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:34:37.249 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:37.322 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:37.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:37.520 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:37.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:37.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:37.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:38.037 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:34:38.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:38.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:37 smithi117 ceph-mon[32835]: pgmap v2324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:38.084 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:37 smithi117 ceph-mon[32835]: osdmap e791: 12 total, 3 up, 11 in 2024-08-06T18:34:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:38.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:37 smithi107 ceph-mon[32381]: pgmap v2324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:38.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:37 smithi107 ceph-mon[32381]: osdmap e791: 12 total, 3 up, 11 in 2024-08-06T18:34:38.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:37 smithi086 ceph-mon[29229]: pgmap v2324: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:38.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:37 smithi086 ceph-mon[29229]: osdmap e791: 12 total, 3 up, 11 in 2024-08-06T18:34:38.329 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:38.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:34:38.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:38.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:38.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:38.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:39.002 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:39.026 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:39.026 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:39.136 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:39.141 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:39.165 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:39.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:39.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:39.431 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:39.431 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:39.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:39.675 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:39.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:39.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:39.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:39.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:39.969 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:39 smithi117 ceph-mon[32835]: pgmap v2326: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:40.079 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:40.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:39 smithi086 ceph-mon[29229]: pgmap v2326: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:40.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:40.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:39 smithi107 ceph-mon[32381]: pgmap v2326: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:40.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:40.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:40.215 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:40.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:40.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:40.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:40.707 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:40.707 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:40.707 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:40.725 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:40.726 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:40.743 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:40.744 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:40.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:40.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:40.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:40.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:41.016 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:41.141 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1038265592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:41.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:41.177 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1038265592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:41.288 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:41.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1038265592' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:41.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:41.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:41.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:41.559 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:41.695 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:41.827 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:41.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:41.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:41.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: pgmap v2327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:41.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.892 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.893 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.893 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:34:41.893 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:41 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:34:41.915 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: pgmap v2327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:41.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:41.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:34:41.916 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:41 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:34:41.916 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:41.916 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:41.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:42.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:42.228 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:42.360 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: pgmap v2327: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:34:42.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:41 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:34:42.496 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:42.632 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:42.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:42.783 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:42.784 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:42.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:42.906 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:42.906 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:42.912 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:34:43.039 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:43.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:42 smithi086 ceph-mon[29229]: osdmap e792: 12 total, 3 up, 11 in 2024-08-06T18:34:43.062 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:42 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:42 smithi117 ceph-mon[32835]: osdmap e792: 12 total, 3 up, 11 in 2024-08-06T18:34:43.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:42 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:43.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:43.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:43.200 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:43.200 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:42 smithi107 ceph-mon[32381]: osdmap e792: 12 total, 3 up, 11 in 2024-08-06T18:34:43.200 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:42 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:34:43.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:43.336 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:43.336 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:43.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:43.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:43.714 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:43.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:43.867 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:43.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:43 smithi086 ceph-mon[29229]: pgmap v2329: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:43 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:44.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:44.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:43 smithi086 ceph-mon[29229]: osdmap e793: 12 total, 3 up, 11 in 2024-08-06T18:34:44.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:44.145 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:43 smithi117 ceph-mon[32835]: pgmap v2329: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:44.145 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:43 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:44.145 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:44.145 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:43 smithi117 ceph-mon[32835]: osdmap e793: 12 total, 3 up, 11 in 2024-08-06T18:34:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:43 smithi107 ceph-mon[32381]: pgmap v2329: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:43 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1264196956' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:34:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:43 smithi107 ceph-mon[32381]: osdmap e793: 12 total, 3 up, 11 in 2024-08-06T18:34:44.254 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:44.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:44.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:44.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:34:44.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:44.785 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:34:44.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:44.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:45.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:45.070 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:45.071 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:45.114 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:45.191 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:45.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:45.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:45.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:45.727 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:45.787 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:45.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:45.888 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:45.888 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:45.888 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:45.888 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:45.910 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:45.910 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:45.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:46.128 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:46.149 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:45 smithi086 ceph-mon[29229]: pgmap v2331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:46.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:45 smithi107 ceph-mon[32381]: pgmap v2331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:46.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:46.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:45 smithi117 ceph-mon[32835]: pgmap v2331: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:46.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:46.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:46.554 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:46.554 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:46.663 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:46.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:46.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:47.065 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:47.073 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:34:47.091 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:46 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:34:47.091 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:46 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:34:47.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:46 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:34:47.094 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:46 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:34:47.095 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:47.095 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:47.201 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:47.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:46 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:34:47.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:46 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:34:47.334 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:47.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:47.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:47.621 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:47.621 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:47.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:47.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:47.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:48.009 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:34:48.126 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:48.189 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:47 smithi107 ceph-mon[32381]: pgmap v2332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:47 smithi107 ceph-mon[32381]: osdmap e794: 12 total, 3 up, 11 in 2024-08-06T18:34:48.262 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:48.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:47 smithi086 ceph-mon[29229]: pgmap v2332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:48.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:47 smithi086 ceph-mon[29229]: osdmap e794: 12 total, 3 up, 11 in 2024-08-06T18:34:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:47 smithi117 ceph-mon[32835]: pgmap v2332: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:47 smithi117 ceph-mon[32835]: osdmap e794: 12 total, 3 up, 11 in 2024-08-06T18:34:48.396 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:48.528 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:48.614 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:34:48.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:34:48.635 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:48.636 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:48.658 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:48.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:48.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:49.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:49.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:49.325 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:49.334 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:49.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:49.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:49.699 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:49.700 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:49.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:49.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:49.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:49.804 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:49.804 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:49.804 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:49.809 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:49.810 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:49.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:49.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:49.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:49.830 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:49.831 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:49.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:49.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:49.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:49.832 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:49.832 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:49.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:50.000 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:50.026 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:49 smithi117 ceph-mon[32835]: pgmap v2334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:50.026 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1105973194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:49 smithi086 ceph-mon[29229]: pgmap v2334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:50.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1105973194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:50.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:50.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:49 smithi107 ceph-mon[32381]: pgmap v2334: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1105973194' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:50.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:50.416 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:50.552 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:50.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:50.709 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:50.709 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:50.819 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:50.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:50.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:51.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.105 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.189 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:51.190 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:34:51.190 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:34:51.213 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:51.349 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:51.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:51.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:51.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:51.640 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:51.749 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:51.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:52.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:52.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:51 smithi086 ceph-mon[29229]: pgmap v2335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:52.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:51 smithi086 ceph-mon[29229]: osdmap e795: 12 total, 3 up, 11 in 2024-08-06T18:34:52.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:52.179 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:51 smithi117 ceph-mon[32835]: pgmap v2335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:52.179 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:51 smithi117 ceph-mon[32835]: osdmap e795: 12 total, 3 up, 11 in 2024-08-06T18:34:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:51 smithi107 ceph-mon[32381]: pgmap v2335: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:52.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:51 smithi107 ceph-mon[32381]: osdmap e795: 12 total, 3 up, 11 in 2024-08-06T18:34:52.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:52.412 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:52.544 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:52.677 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:52.809 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:52.936 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:34:52.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:52.972 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:34:52.972 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:34:53.077 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:53.191 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:34:53.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:53.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:53.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:53.214 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:53.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:53.370 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:34:53.480 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:53.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:53.746 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:53.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:54.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:54.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:53 smithi107 ceph-mon[32381]: pgmap v2337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:53 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:53 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:54.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:53 smithi107 ceph-mon[32381]: osdmap e796: 12 total, 3 up, 11 in 2024-08-06T18:34:54.283 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:54.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:54.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:53 smithi086 ceph-mon[29229]: pgmap v2337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:53 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:53 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:54.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:53 smithi086 ceph-mon[29229]: osdmap e796: 12 total, 3 up, 11 in 2024-08-06T18:34:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:53 smithi117 ceph-mon[32835]: pgmap v2337: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:53 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:34:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:53 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/600890223' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:34:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:53 smithi117 ceph-mon[32835]: osdmap e796: 12 total, 3 up, 11 in 2024-08-06T18:34:54.420 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:54.532 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:34:54.536 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:54.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:54.625 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:54.625 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:54.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:54.823 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:54.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:55.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:55.228 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:55.268 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:55.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:55.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:55.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:55.765 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:55.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:34:55.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:34:56.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:56.058 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:55 smithi107 ceph-mon[32381]: pgmap v2339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:56.058 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:56.058 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:56.168 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:56.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:56.327 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:55 smithi117 ceph-mon[32835]: pgmap v2339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:56.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:55 smithi086 ceph-mon[29229]: pgmap v2339: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:56.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:56.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:56.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:56.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:34:56.842 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:34:56.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:57.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:57.242 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:57.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:57.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:57.399 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:57.433 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:57.433 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:57.509 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:57.640 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:34:57.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:57.645 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:34:57.646 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:34:57.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:57.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:34:57.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:57 smithi117 ceph-mon[32835]: pgmap v2340: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:57.940 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:57 smithi117 ceph-mon[32835]: osdmap e797: 12 total, 3 up, 11 in 2024-08-06T18:34:57.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:57 smithi086 ceph-mon[29229]: pgmap v2340: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:57.942 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:57 smithi086 ceph-mon[29229]: osdmap e797: 12 total, 3 up, 11 in 2024-08-06T18:34:58.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:58.185 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:58.210 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:58.211 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:58.211 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:57 smithi107 ceph-mon[32381]: pgmap v2340: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:58.211 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:57 smithi107 ceph-mon[32381]: osdmap e797: 12 total, 3 up, 11 in 2024-08-06T18:34:58.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:34:58.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:34:58.476 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:34:58.476 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:34:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:34:58.585 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:34:58.715 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:34:58.715 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:34:58.716 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:34:58.720 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:58.731 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:34:58.731 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:34:58.731 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:34:58.731 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:34:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:34:58.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:34:58.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:34:58.732 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:34:58.749 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:34:58.750 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:34:58.856 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:58.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:34:59.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:58 smithi086 ceph-mon[29229]: osdmap e798: 12 total, 3 up, 11 in 2024-08-06T18:34:59.015 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1742048278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:58 smithi117 ceph-mon[32835]: osdmap e798: 12 total, 3 up, 11 in 2024-08-06T18:34:59.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1742048278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:59.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:58 smithi107 ceph-mon[32381]: osdmap e798: 12 total, 3 up, 11 in 2024-08-06T18:34:59.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1742048278' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:34:59.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:34:59.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:59.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:34:59.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:59.800 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:34:59.885 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:34:59.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:34:59.957 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:34:59.957 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:34:59.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:59 smithi107 ceph-mon[32381]: pgmap v2343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:34:59.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:59.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:34:59.958 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:34:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.066 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:59 smithi086 ceph-mon[29229]: pgmap v2343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:34:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:00.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:34:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:00.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:59 smithi117 ceph-mon[32835]: pgmap v2343: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:00.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:34:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:00.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:00.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:00.448 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:00.468 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:00.599 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:00.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:00.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:00.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:00.735 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:00.866 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:01.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:01.055 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:01.135 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:01.266 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:01.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:01.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:01.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:01.291 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:01.402 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:01.534 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:01.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:01.688 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:01.688 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:01.797 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:01.886 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:35:01.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: pgmap v2344: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:01.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: osdmap e799: 12 total, 3 up, 11 in 2024-08-06T18:35:01.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:01 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:02.059 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:02.190 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: pgmap v2344: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: osdmap e799: 12 total, 3 up, 11 in 2024-08-06T18:35:02.212 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:01 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:02.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: pgmap v2344: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:02.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:02.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:02.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:02.294 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2381862819' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:02.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: osdmap e799: 12 total, 3 up, 11 in 2024-08-06T18:35:02.295 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:01 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:02.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:02.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:35:02.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:02.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:02.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:02.624 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:02.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:649,from:6,reservation_nonce:28)", 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:35:02.875 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 526.777592357, 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000147861, 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.876 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098767+0000", 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098768+0000", 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.4e-07 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.877 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098773+0000", 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.9139999999999999e-06 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098783+0000", 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0508e-05 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.878 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098790+0000", 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5409999999999997e-06 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098871+0000", 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1824999999999997e-05 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.879 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098887+0000", 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5617e-05 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.098915+0000", 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7515999999999999e-05 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.880 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.19,from:0'0,to:0'0,epoch:649/121,start:2:98000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 526.77684469600001, 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00013723899999999999, 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.881 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099514+0000", 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.882 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.885 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.885 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.885 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099515+0000", 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6300000000000005e-07 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099522+0000", 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0110000000000003e-06 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.886 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099533+0000", 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1272e-05 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099540+0000", 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.4670000000000001e-06 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.887 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099619+0000", 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9609000000000003e-05 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099628+0000", 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.5110000000000003e-06 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.888 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:26:16.099652+0000", 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3706e-05 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:673,from:2,reservation_nonce:27)", 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 433.44599663499997, 2024-08-06T18:35:02.889 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145129, 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.890 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430364+0000", 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.513e-06 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.891 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430371+0000", 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0190000000000001e-06 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430385+0000", 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.892 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430396+0000", 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0229e-05 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430466+0000", 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0858000000000005e-05 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.893 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430481+0000", 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4596999999999999e-05 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430508+0000", 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.894 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:673/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 433.44538889199998, 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111407, 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.895 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.896 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.897 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430971+0000", 2024-08-06T18:35:02.897 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6599999999999996e-07 2024-08-06T18:35:02.897 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.897 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.897 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430975+0000", 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2889999999999999e-06 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.899 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430986+0000", 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0729e-05 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430992+0000", 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.694e-06 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.900 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431040+0000", 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8418999999999998e-05 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431050+0000", 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431082+0000", 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1754999999999997e-05 2024-08-06T18:35:02.901 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:673,from:2,reservation_nonce:0)", 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 433.44477566400002, 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012740599999999999, 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.902 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.903 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431584+0000", 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.36e-07 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431586+0000", 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0080000000000001e-06 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.904 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431590+0000", 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0090000000000001e-06 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431596+0000", 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2759999999999997e-06 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.905 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431681+0000", 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4974000000000003e-05 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431696+0000", 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4654000000000001e-05 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.906 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431711+0000", 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5248999999999999e-05 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:697,from:2,reservation_nonce:28)", 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 357.65290833400002, 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014356300000000001, 2024-08-06T18:35:02.907 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.908 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223452+0000", 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3510000000000001e-06 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223460+0000", 2024-08-06T18:35:02.909 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9049999999999997e-06 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223475+0000", 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5021999999999999e-05 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223483+0000", 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0350000000000001e-06 2024-08-06T18:35:02.910 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223540+0000", 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7235999999999999e-05 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223561+0000", 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0309000000000002e-05 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.911 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223594+0000", 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3704999999999997e-05 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:697,from:2,reservation_nonce:0)", 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 357.65173495900001, 2024-08-06T18:35:02.912 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110475, 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.913 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.77e-07 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224627+0000", 2024-08-06T18:35:02.914 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.424e-06 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224633+0000", 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2169999999999999e-06 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224638+0000", 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0010000000000001e-06 2024-08-06T18:35:02.915 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224712+0000", 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4124999999999999e-05 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224723+0000", 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0895e-05 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.916 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224735+0000", 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1436e-05 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:699,from:6,reservation_nonce:30)", 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 351.58328079699999, 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022859099999999999, 2024-08-06T18:35:02.917 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.918 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293081+0000", 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4099999999999998e-06 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293094+0000", 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2809999999999999e-05 2024-08-06T18:35:02.919 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293125+0000", 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0998999999999999e-05 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293139+0000", 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4667e-05 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.920 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293256+0000", 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116816 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293272+0000", 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5574e-05 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.921 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293307+0000", 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5315000000000003e-05 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:699,from:6,reservation_nonce:0)", 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 351.58167214299999, 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016018, 2024-08-06T18:35:02.922 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.923 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7e-07 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294707+0000", 2024-08-06T18:35:02.924 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9466e-05 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294716+0000", 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6660000000000003e-06 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294722+0000", 2024-08-06T18:35:02.925 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6489999999999996e-06 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294825+0000", 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102699 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294838+0000", 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3339999999999999e-05 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.926 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294847+0000", 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9900000000000003e-06 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:764,from:2,reservation_nonce:31)", 2024-08-06T18:35:02.927 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 117.036479832, 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136048, 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.928 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839881+0000", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839889+0000", 2024-08-06T18:35:02.929 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2099999999999993e-06 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839905+0000", 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5970000000000001e-05 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839914+0000", 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1839999999999999e-06 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.930 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839972+0000", 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8715999999999998e-05 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839991+0000", 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8578999999999999e-05 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.931 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840015+0000", 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.438e-05 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:764/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:35:02.932 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 117.035925303, 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112552, 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.933 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2399999999999998e-07 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.934 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840438+0000", 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0589999999999996e-06 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840450+0000", 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1161e-05 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.935 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840455+0000", 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3539999999999999e-06 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840503+0000", 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8044000000000003e-05 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.936 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840513+0000", 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0468999999999999e-05 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840546+0000", 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2941000000000002e-05 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.937 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:764,from:2,reservation_nonce:0)", 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 117.035259943, 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115126, 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:02.938 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.939 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5400000000000002e-07 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841101+0000", 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7859999999999999e-06 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.940 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841105+0000", 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8399999999999997e-06 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841110+0000", 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7550000000000002e-06 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:02.941 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841194+0000", 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4648999999999995e-05 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841205+0000", 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0713000000000001e-05 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:02.942 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841214+0000", 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1290000000000002e-06 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:02.943 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:35:02.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:02.944 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:03.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:03.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.068 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:03.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:03.069 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:02 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:03.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:03.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:02 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.178 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:03.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:02 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:03.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:03.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:03.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:03.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:03.575 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:03.606 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:35:03.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:03.842 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:03.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:03.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:03.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:04.003 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:03 smithi117 ceph-mon[32835]: pgmap v2346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:04.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:04.133 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:03 smithi086 ceph-mon[29229]: pgmap v2346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:04.166 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:04.206 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:04.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:03 smithi107 ceph-mon[32381]: pgmap v2346: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:04.246 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:04.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:04.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:04.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:04.791 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:04.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:05.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:05.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:05.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:05.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:05.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:04 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:05.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:05.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:05.593 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:05.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:05.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:05.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:05.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:05.746 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:05.855 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:05.985 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:06.006 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:05 smithi086 ceph-mon[29229]: pgmap v2347: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:06.006 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:06.006 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:06.115 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:06.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:35:06.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:05 smithi107 ceph-mon[32381]: pgmap v2347: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:06.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:06.275 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:05 smithi117 ceph-mon[32835]: pgmap v2347: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:06.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:06.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:06.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:06.566 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:06.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:06.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:06.799 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:06.800 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:06.800 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:06.817 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:06.818 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:06.835 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:06.835 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:06.836 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:06.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:06.949 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/429592955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:06.949 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:06.949 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:07.058 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:07.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/429592955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:07.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:07.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:07.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:07.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:07.354 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/429592955' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:07.447 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:07.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:07.597 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:07.730 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:07.866 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:07.917 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:07.917 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:08.002 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: pgmap v2348: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:35:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:07 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:35:08.135 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:08.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: pgmap v2348: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:08.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.161 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:35:08.162 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:07 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:35:08.176 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:08.177 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: pgmap v2348: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:35:08.177 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:07 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:35:08.210 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:08.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:35:08.245 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:08.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:08.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:08.537 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:08.637 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:08.638 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:08.671 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:08.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:08.695 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:08.695 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:08.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:08.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:08.961 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:08.961 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:09.070 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:09.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:09.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:09.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:09.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:09.473 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:09.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:09.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:09.877 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:09.903 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:09.903 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:10.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:10.038 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:09 smithi117 ceph-mon[32835]: pgmap v2349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:10.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:10.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:09 smithi086 ceph-mon[29229]: pgmap v2349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:09 smithi107 ceph-mon[32381]: pgmap v2349: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:10.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:10.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:10.280 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:10.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:10.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:10.551 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:10.551 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:10.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:35:10.677 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:10.702 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:10.703 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:10.812 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:10.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:10.851 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:10.923 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:10.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:11.081 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:11.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:11.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:11.440 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:11.440 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:11.485 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:11.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:11.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:11.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:11.848 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:11.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:12.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:12.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:11 smithi086 ceph-mon[29229]: pgmap v2350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:12.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:12.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:11 smithi117 ceph-mon[32835]: pgmap v2350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:12.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:11 smithi107 ceph-mon[32381]: pgmap v2350: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:12.279 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:12.410 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:12.541 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:12.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:12.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:12.928 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:35:12.942 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:13.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:13.102 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:13.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:13.285 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:13.346 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:13.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:13.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:13.613 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:13.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:13.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:13.747 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:13.771 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:13.772 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:13.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:14.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:14.030 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:14.038 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:13 smithi107 ceph-mon[32381]: pgmap v2351: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:14.038 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:13 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:14.038 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:13 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:14.038 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:13 smithi107 ceph-mon[32381]: osdmap e800: 12 total, 3 up, 11 in 2024-08-06T18:35:14.148 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:14.182 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:13 smithi117 ceph-mon[32835]: pgmap v2351: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:14.182 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:13 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:14.182 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:13 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:14.182 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:13 smithi117 ceph-mon[32835]: osdmap e800: 12 total, 3 up, 11 in 2024-08-06T18:35:14.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:13 smithi086 ceph-mon[29229]: pgmap v2351: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:14.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:13 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:14.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:13 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4213238347' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:14.266 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:13 smithi086 ceph-mon[29229]: osdmap e800: 12 total, 3 up, 11 in 2024-08-06T18:35:14.284 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:14.416 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:14.534 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:14.534 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:14.534 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:14.534 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:14.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:14.560 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:14.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:35:14.682 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:14.788 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:14.789 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:14.789 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:14.809 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:14.809 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:14.809 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:14.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:14.810 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:14.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:14.829 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:14.830 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:14.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:14.830 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:14.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:14.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:14.831 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:14.951 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:15.031 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:15.087 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:15.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:14 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2599940404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:15.109 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:14 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:14 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2599940404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:15.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:14 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:15.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:15.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:15.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:15.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:15.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:14 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2599940404' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:15.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:14 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:15.484 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:15.509 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:15.509 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:15.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:15.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:35:15.748 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:15.836 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:15.836 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:15.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:16.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:16.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:15 smithi117 ceph-mon[32835]: pgmap v2353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:16.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.039 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:15 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:15 smithi086 ceph-mon[29229]: pgmap v2353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:16.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.111 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:15 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.148 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:15 smithi107 ceph-mon[32381]: pgmap v2353: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:15 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:16.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:16.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:16.547 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:16.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:16.808 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:16.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:16.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:16.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:16.981 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:17.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:17.101 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:17.102 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:17.211 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:17.343 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:17.474 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:17.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:35:17.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:17.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:17.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:17.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:17.872 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:17.891 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:17.892 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:18.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:17 smithi086 ceph-mon[29229]: pgmap v2354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:17 smithi086 ceph-mon[29229]: osdmap e801: 12 total, 3 up, 11 in 2024-08-06T18:35:18.139 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:17 smithi107 ceph-mon[32381]: pgmap v2354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:17 smithi107 ceph-mon[32381]: osdmap e801: 12 total, 3 up, 11 in 2024-08-06T18:35:18.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:17 smithi117 ceph-mon[32835]: pgmap v2354: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:17 smithi117 ceph-mon[32835]: osdmap e801: 12 total, 3 up, 11 in 2024-08-06T18:35:18.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:18.541 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:18.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:18.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:18.947 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:19.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:19.204 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:19.208 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:19.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:19.240 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:19.240 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:19.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:19.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:19.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:19.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:19.622 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:19.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:19.886 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:19.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:35:19.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:19.895 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:35:20.030 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:19 smithi086 ceph-mon[29229]: pgmap v2356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:20.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:19 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:20.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:20.192 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:19 smithi117 ceph-mon[32835]: pgmap v2356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:20.192 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:19 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:19 smithi107 ceph-mon[32381]: pgmap v2356: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:20.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:19 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:20.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:20.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:20.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:20.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:20.570 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:20.594 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:20.594 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:20.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:20.840 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:20.901 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:20.902 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:20.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:21.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:20 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:20 smithi117 ceph-mon[32835]: osdmap e802: 12 total, 3 up, 11 in 2024-08-06T18:35:21.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:20 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:21.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:21.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:20 smithi107 ceph-mon[32381]: osdmap e802: 12 total, 3 up, 11 in 2024-08-06T18:35:21.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:21.281 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:21.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:20 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:21.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3162036289' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:21.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:20 smithi086 ceph-mon[29229]: osdmap e802: 12 total, 3 up, 11 in 2024-08-06T18:35:21.381 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:21.515 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:21.649 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:21.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:21.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:22.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:21 smithi086 ceph-mon[29229]: pgmap v2358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:22.030 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:21 smithi086 ceph-mon[29229]: osdmap e803: 12 total, 3 up, 11 in 2024-08-06T18:35:22.030 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:22.030 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:22.047 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:22.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:22.204 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:21 smithi107 ceph-mon[32381]: pgmap v2358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:22.205 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:21 smithi107 ceph-mon[32381]: osdmap e803: 12 total, 3 up, 11 in 2024-08-06T18:35:22.314 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:22.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:21 smithi117 ceph-mon[32835]: pgmap v2358: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:22.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:21 smithi117 ceph-mon[32835]: osdmap e803: 12 total, 3 up, 11 in 2024-08-06T18:35:22.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:22.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:22.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:35:22.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:22.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:22.950 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:22.988 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:23.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:23.260 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:23.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:23.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:23.663 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:23.797 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:23.825 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:35:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:35:23.825 INFO:teuthology.orchestra.run.smithi086.stdout: "num_ops": 0 2024-08-06T18:35:23.825 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:35:23.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:24.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:24.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:23 smithi086 ceph-mon[29229]: pgmap v2360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:24.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:24.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:24.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:24.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:23 smithi107 ceph-mon[32381]: pgmap v2360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:24.333 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:24.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:23 smithi117 ceph-mon[32835]: pgmap v2360: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:24.428 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:24.428 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:24.429 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:24.445 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:24.446 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:24.446 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:24.446 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:24.446 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:24.463 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:24.463 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:24.465 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:24.466 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:24.466 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:24.466 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:24.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:24.467 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:24.489 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:24.489 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:24.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:24.765 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:24.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:25.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3474292974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:25.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:35:25.173 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:25.198 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3474292974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:25.199 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:25.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3474292974' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:25.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:25.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:25.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:25.577 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:25.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:25.849 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:25.860 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:25.861 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:25.899 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:25.900 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:25.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:26.010 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:25 smithi117 ceph-mon[32835]: pgmap v2361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:26.011 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.011 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:25 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:25 smithi086 ceph-mon[29229]: pgmap v2361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:25 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:26.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:26.120 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:25 smithi107 ceph-mon[32381]: pgmap v2361: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:25 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:26.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:26.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:26.414 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:26.414 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:26.454 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:26.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:26.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:26.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:26.685 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:26.791 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:26.927 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:27.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:27.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:27.211 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:27.211 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:27.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:27.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:27.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:27.592 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:27.592 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:27.617 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:27.617 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:27.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:27.858 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:27.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:27.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:27.900 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:35:27.994 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:28.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:28.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:27 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:35:28.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:27 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:35:28.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:27 smithi107 ceph-mon[32381]: pgmap v2362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:28.154 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:27 smithi107 ceph-mon[32381]: osdmap e804: 12 total, 3 up, 11 in 2024-08-06T18:35:28.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:27 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:35:28.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:27 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:35:28.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:27 smithi086 ceph-mon[29229]: pgmap v2362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:28.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:27 smithi086 ceph-mon[29229]: osdmap e804: 12 total, 3 up, 11 in 2024-08-06T18:35:28.264 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:27 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:35:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:27 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:35:28.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:27 smithi117 ceph-mon[32835]: pgmap v2362: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:28.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:27 smithi117 ceph-mon[32835]: osdmap e804: 12 total, 3 up, 11 in 2024-08-06T18:35:28.398 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:28.436 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:28.436 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:28.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:28.535 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:35:28.535 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:35:28.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:28.797 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:28.900 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:35:28.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:29.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:29.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:29.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:29.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:29.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:29.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:29.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:29.267 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:35:29.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:29.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:29.607 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:29.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:29.714 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:29.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:29.883 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:29.931 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:30.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:29 smithi117 ceph-mon[32835]: pgmap v2364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:29 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:30.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:29 smithi117 ceph-mon[32835]: osdmap e805: 12 total, 3 up, 11 in 2024-08-06T18:35:30.149 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:30.172 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:30.172 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:30.172 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:29 smithi086 ceph-mon[29229]: pgmap v2364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:30.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:29 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:30.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:30.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:29 smithi086 ceph-mon[29229]: osdmap e805: 12 total, 3 up, 11 in 2024-08-06T18:35:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:29 smithi107 ceph-mon[32381]: pgmap v2364: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:29 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/450851898' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:29 smithi107 ceph-mon[32381]: osdmap e805: 12 total, 3 up, 11 in 2024-08-06T18:35:30.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:30.426 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:30.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:30.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:30.831 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:30.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:31.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:31.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:31.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:31.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:31.597 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:31.598 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:31.598 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:31.608 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:31.618 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:31.619 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:31.638 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:31.639 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:31.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:31.812 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:31.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:31.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:35:31.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:31 smithi107 ceph-mon[32381]: pgmap v2366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:31.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3352586422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:32.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:32.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:32.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:31 smithi086 ceph-mon[29229]: pgmap v2366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:32.235 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3352586422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:32.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:32.371 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:31 smithi117 ceph-mon[32835]: pgmap v2366: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 335 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:32.371 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3352586422' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:32.481 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:32.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:32.562 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:32.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:32.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:32.885 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:32.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:32.905 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:33.010 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:33.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:33.157 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:33.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.179 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:32 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:32 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.288 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:33.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:32 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:33.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:33.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:33.575 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:33.575 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:33.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:33.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:33.954 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:35:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:35:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:35:33.954 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:673,from:0,reservation_nonce:0)", 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 464.52509467800002, 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00017925499999999999, 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.955 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430573+0000", 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430574+0000", 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8299999999999996e-07 2024-08-06T18:35:33.956 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430577+0000", 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.864e-06 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430587+0000", 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0581000000000001e-05 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.957 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430593+0000", 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7060000000000001e-06 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430636+0000", 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2914000000000003e-05 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430649+0000", 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3098999999999999e-05 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.958 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430752+0000", 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010330799999999999 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 464.52444677199998, 2024-08-06T18:35:33.959 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6106999999999996e-05, 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431221+0000", 2024-08-06T18:35:33.960 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431222+0000", 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.4899999999999995e-07 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431225+0000", 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2949999999999998e-06 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.961 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431233+0000", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8660000000000006e-06 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431237+0000", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.1880000000000004e-06 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431266+0000", 2024-08-06T18:35:33.962 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.8912e-05 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431271+0000", 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.5330000000000002e-06 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431287+0000", 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5764e-05 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.963 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:697,from:0,reservation_nonce:0)", 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 388.73202548900002, 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000184933, 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.964 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223642+0000", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223643+0000", 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0699999999999996e-07 2024-08-06T18:35:33.965 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223646+0000", 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5220000000000002e-06 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223655+0000", 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1509999999999993e-06 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.966 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223660+0000", 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3759999999999999e-06 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223696+0000", 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223709+0000", 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3013999999999999e-05 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.967 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223827+0000", 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000118313 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:35:33.968 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 388.73141749199999, 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.7805000000000005e-05, 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.969 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224250+0000", 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224251+0000", 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8699999999999995e-07 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224254+0000", 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.7010000000000001e-06 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.970 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224261+0000", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8439999999999998e-06 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224266+0000", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3579999999999996e-06 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224302+0000", 2024-08-06T18:35:33.971 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6529999999999998e-05 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224310+0000", 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.8839999999999994e-06 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224328+0000", 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7901e-05 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.972 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,GRANT,e:764,from:0,reservation_nonce:0)", 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 148.11559816299999, 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016942300000000001, 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:35:33.973 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840070+0000", 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1999999999999998e-07 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.974 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840074+0000", 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1690000000000001e-06 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840082+0000", 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4290000000000001e-06 2024-08-06T18:35:33.975 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840088+0000", 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7710000000000003e-06 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840124+0000", 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5849999999999997e-05 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.976 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840136+0000", 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2211e-05 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840239+0000", 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000103173 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.977 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "rep_scrubmap(2.1 e121 from shard 0)", 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 148.11497834100001, 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102233, 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.978 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840689+0000", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840690+0000", 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9699999999999995e-07 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.979 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840695+0000", 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2869999999999998e-06 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840703+0000", 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1419999999999993e-06 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:35:33.980 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840708+0000", 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.1189999999999996e-06 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840751+0000", 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.3139999999999997e-05 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.981 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840775+0000", 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3730000000000001e-05 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:35:33.982 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840792+0000", 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7017999999999999e-05 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:35:33.983 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:35:33.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:33.985 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:33 smithi086 ceph-mon[29229]: pgmap v2367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:34.009 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:33 smithi107 ceph-mon[32381]: pgmap v2367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:34.118 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:34.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:34.314 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:33 smithi117 ceph-mon[32835]: pgmap v2367: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:34.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:34.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:34.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:34.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:34.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:34 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:34.789 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:34.859 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:35:34.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:34.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:34.966 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:35.011 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:35:35.060 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:35.120 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:35.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:35.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:35.458 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:35.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:35.728 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:35.862 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:35.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:36.097 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:35 smithi086 ceph-mon[29229]: pgmap v2368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:36.126 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:36.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:35 smithi107 ceph-mon[32381]: pgmap v2368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:36.262 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:36.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:35 smithi117 ceph-mon[32835]: pgmap v2368: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:36.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:36.525 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:36.634 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:36.634 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:36.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:36.730 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:36.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:36.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:36.941 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:36.941 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:37.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:37.180 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:37.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:37.384 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:37.385 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:35:37.446 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:37.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:37.677 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:37.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:37.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:37.771 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:37.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:38.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:38.024 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:37 smithi086 ceph-mon[29229]: pgmap v2369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:38.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:37 smithi107 ceph-mon[32381]: pgmap v2369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:38.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:38.233 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:38.233 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:38.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:38.250 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:38.250 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:38.250 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:38.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:38.251 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:38.263 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:38.268 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:38.269 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:38.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:38.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:38.269 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:38.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:38.285 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:38.285 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:37 smithi117 ceph-mon[32835]: pgmap v2369: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:38.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:38.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:38.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:38.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:38.616 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:38.658 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:38.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:38.918 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:35:38.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:38.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:38.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:39.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:39.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:39.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2352207241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:39.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.084 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:38 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2352207241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:38 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.194 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:39.227 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:39.227 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2352207241' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:39.227 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.227 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:38 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:39.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:39.335 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:39.335 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:39.358 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:39.456 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:39.592 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:39.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:39.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:39.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:39.857 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:39.941 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:35:39.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:40.124 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:40.189 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:39 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:39 smithi107 ceph-mon[32381]: pgmap v2370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:39 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:39 smithi107 ceph-mon[32381]: osdmap e806: 12 total, 3 up, 11 in 2024-08-06T18:35:40.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:40.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:39 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:40.281 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:39 smithi117 ceph-mon[32835]: pgmap v2370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:40.281 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:39 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:40.281 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:40.281 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:39 smithi117 ceph-mon[32835]: osdmap e806: 12 total, 3 up, 11 in 2024-08-06T18:35:40.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:40.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:40.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:39 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:40.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:39 smithi086 ceph-mon[29229]: pgmap v2370: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:40.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:39 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:35:40.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/43777119' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:40.367 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:39 smithi086 ceph-mon[29229]: osdmap e806: 12 total, 3 up, 11 in 2024-08-06T18:35:40.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:40.522 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:40.563 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:40.563 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:40 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:40.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:40.790 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:40.926 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:40.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:35:41.059 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:41.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:41.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:41.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:41.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:41.195 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:41.249 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:41.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:41.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:41.596 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:41.681 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:35:41.708 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:41.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:41.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:41.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:42.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:42.145 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:42.146 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:42.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:41 smithi086 ceph-mon[29229]: pgmap v2372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:42.146 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:41 smithi086 ceph-mon[29229]: osdmap e807: 12 total, 3 up, 11 in 2024-08-06T18:35:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:41 smithi107 ceph-mon[32381]: pgmap v2372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:41 smithi107 ceph-mon[32381]: osdmap e807: 12 total, 3 up, 11 in 2024-08-06T18:35:42.254 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:42.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:41 smithi117 ceph-mon[32835]: pgmap v2372: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:42.279 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:41 smithi117 ceph-mon[32835]: osdmap e807: 12 total, 3 up, 11 in 2024-08-06T18:35:42.347 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:42.348 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:35:42.388 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:42.411 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:42.411 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:42.421 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:42.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:42.598 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:42.653 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:42.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:42.922 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:43.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:43.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:43.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:43.244 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:43.321 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:43.455 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:43.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:43.594 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:43.594 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:43.594 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:43.611 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:43.612 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:43.629 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:43.631 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:43.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:43.860 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:43.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:44.098 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:44.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:44.151 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:43 smithi086 ceph-mon[29229]: pgmap v2374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:44.151 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2718801521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:43 smithi107 ceph-mon[32381]: pgmap v2374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2718801521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:44.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:43 smithi117 ceph-mon[32835]: pgmap v2374: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2718801521' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:44.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:44.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:44.547 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:44.547 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:44.656 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:44.752 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:44.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:35:44.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:44.920 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:44.921 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:44.924 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:35:44.927 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:44.950 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:44.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:44.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:44.951 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:44.999 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:45.059 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:45.189 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:45.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.212 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:45.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:45.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:45.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:35:45.266 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:45.267 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:45.322 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:45.457 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:45.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:45.593 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:45.593 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:45.609 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:45.609 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:45.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:45.853 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:45.984 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:46.113 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:45 smithi117 ceph-mon[32835]: pgmap v2375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:45 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:46.125 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:46.126 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:45 smithi117 ceph-mon[32835]: osdmap e808: 12 total, 3 up, 11 in 2024-08-06T18:35:46.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:46.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:45 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:46.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:45 smithi107 ceph-mon[32381]: pgmap v2375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:46.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:45 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:46.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:46.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:45 smithi107 ceph-mon[32381]: osdmap e808: 12 total, 3 up, 11 in 2024-08-06T18:35:46.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:46.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:45 smithi086 ceph-mon[29229]: pgmap v2375: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:46.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:45 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:46.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3208496433' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:35:46.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:45 smithi086 ceph-mon[29229]: osdmap e808: 12 total, 3 up, 11 in 2024-08-06T18:35:46.380 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:46.512 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:46.587 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:46.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:46.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:46.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:46.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:46.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:47.049 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:47.074 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:47.075 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:46 smithi117 ceph-mon[32835]: osdmap e809: 12 total, 3 up, 11 in 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:47.174 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:46 smithi107 ceph-mon[32381]: osdmap e809: 12 total, 3 up, 11 in 2024-08-06T18:35:47.183 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:47.201 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:47.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:35:47.268 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:35:47.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:35:47.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:35:47.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:35:47.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:35:47.291 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:35:47.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:35:47.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:35:47.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:46 smithi086 ceph-mon[29229]: osdmap e809: 12 total, 3 up, 11 in 2024-08-06T18:35:47.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:47.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:47.503 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:47.519 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:47.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:47.597 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:47.598 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:47.707 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:47.837 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:47.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:47.859 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:47.860 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:47.860 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:47.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:48.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:35:48.098 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:48.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:48.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:48.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:48.364 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:48.498 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:48.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:48.758 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:48.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:48.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:48.894 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:49.021 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:49.027 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:49.128 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:49.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:49.129 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:49 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:49.146 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:49.147 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:49.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:49.293 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:49.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:49.457 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:49.458 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:49.458 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:49.475 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:49.476 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:49.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:49.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:49.476 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:49.492 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:49.492 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:49.494 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:49.495 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:49.495 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:49.496 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:49.496 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:49.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:49 smithi117 ceph-mon[32835]: pgmap v2379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:49.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3587593942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:49.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:49.756 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:49 smithi107 ceph-mon[32381]: pgmap v2379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:49.757 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3587593942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:49.785 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:49.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:35:49.829 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:49.854 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:49.854 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:49.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:49 smithi086 ceph-mon[29229]: pgmap v2379: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:49.854 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3587593942' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:49.960 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:50.034 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:50.093 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:50.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:50.359 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:50.494 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:50.577 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:50.578 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:50.625 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:50 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:50 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:35:50.712 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:50 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:35:50.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:50.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:50 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:50 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:35:50.781 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:50 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:35:50.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:50 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:50 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:35:50.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:50 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:35:50.891 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:51.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:51.155 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:51.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:51.422 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:51.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:51.565 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:35:51.565 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:35:51.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:51.717 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:51 smithi117 ceph-mon[32835]: pgmap v2380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:51.717 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:51 smithi117 ceph-mon[32835]: osdmap e810: 12 total, 3 up, 11 in 2024-08-06T18:35:51.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:51.849 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:51 smithi107 ceph-mon[32381]: pgmap v2380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:51.849 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:51 smithi107 ceph-mon[32381]: osdmap e810: 12 total, 3 up, 11 in 2024-08-06T18:35:51.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:51 smithi086 ceph-mon[29229]: pgmap v2380: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:51.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:51 smithi086 ceph-mon[29229]: osdmap e810: 12 total, 3 up, 11 in 2024-08-06T18:35:51.900 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:35:51.958 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:52.089 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:52.152 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:52.152 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:51 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:52.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:35:52.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:52.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:52.428 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:52.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:52.500 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:52.500 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:52.579 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:35:52.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:52.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:52.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:52 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:52.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:52 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:52.652 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:52 smithi117 ceph-mon[32835]: osdmap e811: 12 total, 3 up, 11 in 2024-08-06T18:35:52.755 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:52.755 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:52 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:52.755 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:52 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:52.755 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:52 smithi107 ceph-mon[32381]: osdmap e811: 12 total, 3 up, 11 in 2024-08-06T18:35:52.761 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:52.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:35:52.783 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:52 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:52.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:52 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/776174388' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:35:52.784 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:52 smithi086 ceph-mon[29229]: osdmap e811: 12 total, 3 up, 11 in 2024-08-06T18:35:52.834 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:52.893 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:53.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:53.162 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:53.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:53.428 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:53.560 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:53.690 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:53.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:53.781 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:53.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:53.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:53.978 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:35:53.978 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:35:53.983 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:53 smithi117 ceph-mon[32835]: pgmap v2383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:54.084 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:53 smithi086 ceph-mon[29229]: pgmap v2383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:54.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:54.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:53 smithi107 ceph-mon[32381]: pgmap v2383: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:54.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:54.223 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:54.315 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:54.316 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:54.332 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:54.332 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:54.333 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:54.350 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:54.350 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:54.350 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:54.351 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:54.351 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:54.352 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:54.352 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:54.352 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:54.357 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:54.491 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:54.623 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:54.635 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:35:54.640 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:35:54.755 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:54.883 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:54.887 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:54.917 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:54 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/41309953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:54.917 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:54.917 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:54 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:55.015 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:35:55.016 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:35:55.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:55.024 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:55.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:35:55.176 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:54 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/41309953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:55.177 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:55.177 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:54 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:54 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/41309953' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:55.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:54 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:55.285 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:55.482 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:55.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:55.748 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:55.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:55.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:55.913 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:55 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:56.018 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:56.051 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:56.051 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:56.051 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:55 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:56.051 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:55 smithi107 ceph-mon[32381]: pgmap v2384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:56.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:56.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:55 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:56.178 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:55 smithi117 ceph-mon[32835]: pgmap v2384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:56.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:55 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:35:56.204 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:55 smithi086 ceph-mon[29229]: pgmap v2384: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:56.287 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "size": 20, 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 600, 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [ 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:649,from:0,reservation_nonce:0)", 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 580.29117260500004, 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00021522199999999999, 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:35:56.389 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099043+0000", 2024-08-06T18:35:56.390 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099044+0000", 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.719e-06 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099049+0000", 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.4410000000000003e-06 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.391 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099061+0000", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.2422999999999999e-05 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099070+0000", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.6349999999999995e-06 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099113+0000", 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 4.3393999999999998e-05 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.392 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099128+0000", 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5233e-05 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099258+0000", 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00012937699999999999 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.393 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 580.29033580400005, 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.4050999999999999e-05, 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:35:56.394 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099879+0000", 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099880+0000", 2024-08-06T18:35:56.395 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 7.5199999999999996e-07 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099886+0000", 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.1040000000000003e-06 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099896+0000", 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.8789999999999998e-06 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.396 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099901+0000", 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.2970000000000003e-06 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099938+0000", 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.6542999999999999e-05 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:35:56.397 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099944+0000", 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.4779999999999996e-06 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:26:16.099963+0000", 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.8998e-05 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.398 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "MOSDScrubReserve(2.19,GRANT,e:699,from:0,reservation_nonce:0)", 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 405.096797041, 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00026914199999999999, 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.399 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293418+0000", 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293420+0000", 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.9920000000000002e-06 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:35:56.400 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293427+0000", 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 6.6719999999999998e-06 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293442+0000", 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.5124e-05 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293453+0000", 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.1022000000000001e-05 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.401 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293534+0000", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 8.0717999999999995e-05 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293552+0000", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 1.7982000000000002e-05 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.293687+0000", 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0.00013563200000000001 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.402 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "description": "rep_scrubmap(2.19 e121 from shard 0)", 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "initiated_at": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "age": 405.09598962000001, 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 9.9434999999999999e-05, 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "continuous": false, 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "type_data": { 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "flag_point": "started", 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: "events": [ 2024-08-06T18:35:56.403 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "initiated", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "header_read", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 0 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "throttled", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294226+0000", 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.0900000000000002e-07 2024-08-06T18:35:56.404 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "all_read", 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294229+0000", 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.9280000000000002e-06 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "dispatched", 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294255+0000", 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.6409000000000001e-05 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.405 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "queued_for_pg", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294261+0000", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.6300000000000003e-06 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "reached_pg", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294298+0000", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 3.7233000000000001e-05 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "started", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294304+0000", 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 5.7960000000000001e-06 2024-08-06T18:35:56.406 INFO:teuthology.orchestra.run.smithi107.stdout: }, 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: { 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: "event": "done", 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: "time": "2024-08-06T18:29:11.294325+0000", 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: "duration": 2.0930000000000001e-05 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: } 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout: ] 2024-08-06T18:35:56.407 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:35:56.421 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:56.553 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:56.688 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:56.710 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:56.710 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:56 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:56.819 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:56.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:56.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:56.928 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:35:56.950 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:56.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:57.017 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:35:57.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:57.102 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:57.102 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:57.102 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:57.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:57.103 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:57.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:35:57.211 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:35:57.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:57.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:57.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:35:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:35:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:35:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:35:57.377 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:35:57.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:35:57.603 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:57.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:35:57.804 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:35:57.879 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:35:57.916 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:57.916 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:57.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:57 smithi086 ceph-mon[29229]: pgmap v2385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:57.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:57 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:57.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:57 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:57.926 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:57 smithi086 ceph-mon[29229]: osdmap e812: 12 total, 3 up, 11 in 2024-08-06T18:35:58.008 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:57 smithi117 ceph-mon[32835]: pgmap v2385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:57 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:57 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:57 smithi117 ceph-mon[32835]: osdmap e812: 12 total, 3 up, 11 in 2024-08-06T18:35:58.139 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:58.168 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:57 smithi107 ceph-mon[32381]: pgmap v2385: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:35:58.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:57 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:35:58.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:57 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2799239543' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:35:58.169 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:57 smithi107 ceph-mon[32381]: osdmap e812: 12 total, 3 up, 11 in 2024-08-06T18:35:58.271 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:58.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:58.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:58.429 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:58.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:58.668 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:58.686 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:58.687 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:58 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:58.803 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:35:58.935 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:59.036 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:35:59.070 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:35:59.201 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:35:59.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:35:59.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:35:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:35:59.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:59.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:35:59.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:35:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:35:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:35:59.605 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:35:59.606 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:35:59.607 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:35:59.684 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:35:59.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:35:59.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:35:59.728 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:35:59.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:35:59.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:35:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:35:59.871 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:35:59.936 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:36:00.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:00.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:00.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:59 smithi107 ceph-mon[32381]: pgmap v2387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:00.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:59 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:36:00.171 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:59 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:36:00.172 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:35:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2824777902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:00.222 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:59 smithi086 ceph-mon[29229]: pgmap v2387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:00.222 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:59 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:36:00.222 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:59 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:36:00.222 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:35:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2824777902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:00.274 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:00.304 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:59 smithi117 ceph-mon[32835]: pgmap v2387: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:59 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:36:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:59 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:36:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:35:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2824777902' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:00.381 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:00.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:00.502 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:00.502 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:00.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:00.667 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:00.802 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:00.933 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:00.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:00.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:00.956 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:00 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.065 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:01.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.087 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:00 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:01.087 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:00 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:01.195 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:01.326 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:01.456 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:01.475 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:01.586 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:01.718 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:01.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:01.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:01.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:01.989 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:02.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:01 smithi107 ceph-mon[32381]: pgmap v2388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:02.044 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:01 smithi107 ceph-mon[32381]: osdmap e813: 12 total, 3 up, 11 in 2024-08-06T18:36:02.072 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:02.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:36:02.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:01 smithi117 ceph-mon[32835]: pgmap v2388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:02.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:01 smithi117 ceph-mon[32835]: osdmap e813: 12 total, 3 up, 11 in 2024-08-06T18:36:02.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:02.306 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:36:02.323 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:36:02.327 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:01 smithi086 ceph-mon[29229]: pgmap v2388: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:02.327 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:01 smithi086 ceph-mon[29229]: osdmap e813: 12 total, 3 up, 11 in 2024-08-06T18:36:02.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:02.327 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:02.328 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:02.328 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:02.391 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:02.527 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:02.551 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:02.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:02.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:02.926 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:02.926 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:02.932 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:36:02.937 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:03.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:03.090 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2162367753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.090 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:02 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.090 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:03.090 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:03.199 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:03.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:03.354 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2162367753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.354 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:02 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2162367753' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:02 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:03.392 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:03.392 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:03.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:03.595 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:03.726 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:03.859 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:03.912 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:03 smithi107 ceph-mon[32381]: pgmap v2390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:03 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:03 smithi107 ceph-mon[32381]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:03.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:03 smithi107 ceph-mon[32381]: osdmap e814: 12 total, 3 up, 11 in 2024-08-06T18:36:03.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:04.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:04.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:04.039 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:04.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:04.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:03 smithi086 ceph-mon[29229]: pgmap v2390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:04.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:03 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:04.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:03 smithi086 ceph-mon[29229]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:04.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:03 smithi086 ceph-mon[29229]: osdmap e814: 12 total, 3 up, 11 in 2024-08-06T18:36:04.126 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:04.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:03 smithi117 ceph-mon[32835]: pgmap v2390: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:03 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:03 smithi117 ceph-mon[32835]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:03 smithi117 ceph-mon[32835]: osdmap e814: 12 total, 3 up, 11 in 2024-08-06T18:36:04.386 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:04.499 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:04.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:36:04.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:04.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:04.749 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:04.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:04.866 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:04.866 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:04.867 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:04.883 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:04.883 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:04.884 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:04.902 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:04.903 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:04.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:04.931 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:04.932 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:05.045 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:05.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1228250403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:05.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:05.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1228250403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:05.305 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:05.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1228250403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:05.435 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:05.565 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:05.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:05.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:05.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:05.837 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:05.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:05.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:05 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:05.908 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:36:05.968 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:05.994 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:05.994 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:05.994 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: pgmap v2392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:05.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:05.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:05.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:05.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:36:05.995 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:05 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:36:06.094 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: pgmap v2392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:06.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:36:06.095 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:05 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:36:06.103 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:06.154 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:06.234 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:06.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: pgmap v2392: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:06.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.256 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:06.257 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:36:06.257 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:05 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:36:06.320 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:06.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:06.496 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:06.628 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:06.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:06.894 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:06.911 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:06.912 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:36:07.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:07.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:06 smithi117 ceph-mon[32835]: osdmap e815: 12 total, 3 up, 11 in 2024-08-06T18:36:07.158 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:07.208 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:07.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:06 smithi107 ceph-mon[32381]: osdmap e815: 12 total, 3 up, 11 in 2024-08-06T18:36:07.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:07.305 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:06 smithi086 ceph-mon[29229]: osdmap e815: 12 total, 3 up, 11 in 2024-08-06T18:36:07.305 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:07.305 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:07.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:07.560 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:07.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:36:07.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:07.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:07.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:07.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:07.909 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:07.964 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:08.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:07 smithi107 ceph-mon[32381]: pgmap v2394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:08.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:08.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:08.227 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:08.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:07 smithi086 ceph-mon[29229]: pgmap v2394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:08.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:08.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:08.313 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:07 smithi117 ceph-mon[32835]: pgmap v2394: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:08.363 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:08.498 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:08.637 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:08.663 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:08.663 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:08.773 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:08.862 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:08.906 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:08.951 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:36:09.038 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:09.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:08 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:09.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:09.116 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:09.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:09.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:08 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:09.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:09.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:09 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:09.302 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:08 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:09.304 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:09.388 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:09.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:36:09.434 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:09.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:09.451 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:09.564 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:09.637 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:09.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:09.775 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:09.776 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:09.792 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:09.793 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:09.810 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:09.811 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:09.830 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:09.964 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: pgmap v2395: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: osdmap e816: 12 total, 3 up, 11 in 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2739451853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:09.988 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:10.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: pgmap v2395: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:10.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:10.082 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:10.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: osdmap e816: 12 total, 3 up, 11 in 2024-08-06T18:36:10.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2739451853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:10.083 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:10.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: pgmap v2395: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3100717367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: osdmap e816: 12 total, 3 up, 11 in 2024-08-06T18:36:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2739451853' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:10.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:10.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:10.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:10.346 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:10.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:10.494 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:10.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:10.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:10.624 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:10.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:10.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:11.005 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:11.006 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:11.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:11.045 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.045 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.062 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.063 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:11.155 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:11.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:11.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:11.420 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:11.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:11.644 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:11.645 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:36:11.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:11.817 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:11.891 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:11.952 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:12.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:12.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:11 smithi086 ceph-mon[29229]: pgmap v2397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:12.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:11 smithi086 ceph-mon[29229]: osdmap e817: 12 total, 3 up, 11 in 2024-08-06T18:36:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:11 smithi107 ceph-mon[32381]: pgmap v2397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:11 smithi107 ceph-mon[32381]: osdmap e817: 12 total, 3 up, 11 in 2024-08-06T18:36:12.214 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:12.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:11 smithi117 ceph-mon[32835]: pgmap v2397: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:12.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:11 smithi117 ceph-mon[32835]: osdmap e817: 12 total, 3 up, 11 in 2024-08-06T18:36:12.349 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:12.484 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:12.502 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:36:12.528 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:12.528 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:12.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:12.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:12.762 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:12.882 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:13.007 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:36:13.013 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:13.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:13.273 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:13.403 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:13.426 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:13.436 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:13.536 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:13.676 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:13.697 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:13.698 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:13.806 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:13.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:13.957 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:13.957 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:14.006 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:14.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:36:14.069 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:13 smithi117 ceph-mon[32835]: pgmap v2399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:14.205 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:13 smithi107 ceph-mon[32381]: pgmap v2399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:14.219 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:13 smithi086 ceph-mon[29229]: pgmap v2399: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:14.219 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:14.220 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:14.253 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:14.340 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:14.470 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:14.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:14.492 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:14.600 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:14.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:14.731 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:14.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:14.753 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:14.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:14.998 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:15.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:15.008 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:15.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:15.215 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:15.215 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:15.263 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:15.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:15.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:15.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:15.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:15.592 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:15.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:15.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:15.924 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:15.990 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:15.990 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:15.991 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:16.007 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:16.007 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:16.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:16.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:16.008 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:16.024 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:16.025 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:16.025 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:15 smithi086 ceph-mon[29229]: pgmap v2400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:16.025 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:16.026 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:16.026 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:16.026 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:16.026 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:16.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:16.027 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:16.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:16.109 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:16.109 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:36:16.193 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:16.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:15 smithi107 ceph-mon[32381]: pgmap v2400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:16.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:16.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:16.354 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:15 smithi117 ceph-mon[32835]: pgmap v2400: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:16.462 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:16.594 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:16.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:16.861 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:16.882 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:16.882 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:16.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:16.946 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:16.964 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:36:16.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:17.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2101996842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:17.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:16 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.020 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:17.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2101996842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:17.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:16 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.114 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:17.129 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:17.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:17.287 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:17.287 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:17.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2101996842' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:17.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:16 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:17.331 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:17.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:17.529 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:17.659 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:17.792 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:17.886 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:17.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:18.061 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:18.193 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:18.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:17 smithi107 ceph-mon[32381]: pgmap v2401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:17 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:18.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:17 smithi107 ceph-mon[32381]: osdmap e818: 12 total, 3 up, 11 in 2024-08-06T18:36:18.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:17 smithi117 ceph-mon[32835]: pgmap v2401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:18.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:17 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:18.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:18.217 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:17 smithi117 ceph-mon[32835]: osdmap e818: 12 total, 3 up, 11 in 2024-08-06T18:36:18.324 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:18.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:17 smithi086 ceph-mon[29229]: pgmap v2401: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:18.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:17 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:18.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3453296852' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:18.348 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:17 smithi086 ceph-mon[29229]: osdmap e818: 12 total, 3 up, 11 in 2024-08-06T18:36:18.456 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:18.463 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:18.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:36:18.589 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:18.711 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:18.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:18.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:18.947 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:36:18.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:19.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:19.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:19.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:19.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:19.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:19.203 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:19.255 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:19.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:19.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:19.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:19.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:19.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:36:19.777 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:19.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:20.016 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:19 smithi117 ceph-mon[32835]: pgmap v2403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:20.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:20.062 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:19 smithi107 ceph-mon[32381]: pgmap v2403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:20.063 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:20.063 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:19 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:20.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:20.216 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:20.303 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:20.325 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:19 smithi086 ceph-mon[29229]: pgmap v2403: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:20.326 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:20.326 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:20.434 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:20.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:20.564 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:20.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:20.586 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:20.694 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:20.762 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:20.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:36:20.826 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:20.911 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:20.911 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:20.911 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:20.912 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:20.928 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:20.929 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:20.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:20.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:20.929 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:20.947 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:20.948 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:20.948 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:20.948 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:20.948 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:20.948 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:20.949 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:20.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:21.007 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:21.090 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3176519155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:21.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:21.215 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3176519155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:21.215 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:21.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:21.216 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:21.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:21.243 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3176519155' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:21.243 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:21.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:21.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:21.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:21.745 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:21.880 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:22.015 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:21 smithi117 ceph-mon[32835]: pgmap v2404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:21 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:21 smithi117 ceph-mon[32835]: osdmap e819: 12 total, 3 up, 11 in 2024-08-06T18:36:22.147 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:22.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:21 smithi107 ceph-mon[32381]: pgmap v2404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:21 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:21 smithi107 ceph-mon[32381]: osdmap e819: 12 total, 3 up, 11 in 2024-08-06T18:36:22.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:22.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:21 smithi086 ceph-mon[29229]: pgmap v2404: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:22.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:21 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:22.344 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:21 smithi086 ceph-mon[29229]: osdmap e819: 12 total, 3 up, 11 in 2024-08-06T18:36:22.371 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:22.374 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:22.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:22.501 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:22.547 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:22.679 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:22.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:22.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:22.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:22.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:23.087 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:23.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:23.094 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:23.198 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:23.199 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:36:23.219 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:23.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:23.448 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:23.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:23.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:23.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:23.785 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:23.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:23.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:23.866 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:23.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:23.983 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:36:24.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:23 smithi117 ceph-mon[32835]: pgmap v2406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:24.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:24.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:24.315 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:24.340 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:23 smithi107 ceph-mon[32381]: pgmap v2406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:24.340 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:24.340 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:24.340 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:24.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:23 smithi086 ceph-mon[29229]: pgmap v2406: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:24.343 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:24.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:36:24.450 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:24.580 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:24.606 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:24.606 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:24.715 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:24.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:24.858 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:24.898 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:24.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:24 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:24 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:25.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:24 smithi086 ceph-mon[29229]: osdmap e820: 12 total, 3 up, 11 in 2024-08-06T18:36:25.116 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:24 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:24 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:25.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:24 smithi117 ceph-mon[32835]: osdmap e820: 12 total, 3 up, 11 in 2024-08-06T18:36:25.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:25.382 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:25.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:24 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:25.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:24 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2450557317' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:25.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:24 smithi107 ceph-mon[32381]: osdmap e820: 12 total, 3 up, 11 in 2024-08-06T18:36:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:25.477 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:36:25.512 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:25.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:25.723 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:25.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:25.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:25.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:25.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:26.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:26.068 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:26 smithi107 ceph-mon[32381]: pgmap v2408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:26 smithi117 ceph-mon[32835]: pgmap v2408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:26.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:26.270 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:26 smithi086 ceph-mon[29229]: pgmap v2408: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:26.270 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:26.271 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:26.309 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:26.441 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:26.571 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:26.617 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:26.618 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:26.618 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:26.634 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:26.635 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:26.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:26.635 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:26.636 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:26.653 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:26.654 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:26.654 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:26.654 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:26.655 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:26.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:26.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:36:26.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:26.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:26.841 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:26.972 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:27.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:27.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1730047685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:27.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:27 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:27.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:27 smithi086 ceph-mon[29229]: osdmap e821: 12 total, 3 up, 11 in 2024-08-06T18:36:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:27.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1730047685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:27 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:27.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:27 smithi107 ceph-mon[32381]: osdmap e821: 12 total, 3 up, 11 in 2024-08-06T18:36:27.235 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:27.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1730047685' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:27.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:27 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:27.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:27 smithi117 ceph-mon[32835]: osdmap e821: 12 total, 3 up, 11 in 2024-08-06T18:36:27.301 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:27.368 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:27.396 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:27.397 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:27.499 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:27.508 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:27.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:27.652 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:27.652 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:27.767 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:27.873 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:27.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:36:27.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:28.032 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:28 smithi117 ceph-mon[32835]: pgmap v2409: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:28 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.133 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:28.163 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:28.297 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:28.319 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:28 smithi086 ceph-mon[29229]: pgmap v2409: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:28.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:28 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.428 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:28 smithi107 ceph-mon[32381]: pgmap v2409: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.453 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:28 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:28.563 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:28.696 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:28.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:28.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:28.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:28.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:28.984 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:29.093 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:29.225 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:29.355 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:29.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:36:29.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:29.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:29.674 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:29.714 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:29.750 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:29.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:29.796 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:29.880 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:30.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:30.012 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:36:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:30 smithi086 ceph-mon[29229]: pgmap v2411: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:30 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:30 smithi117 ceph-mon[32835]: pgmap v2411: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:30.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:30 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:30.146 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:30.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:30.319 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:30.320 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:36:30.412 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:30.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:30 smithi107 ceph-mon[32381]: pgmap v2411: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:30.435 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:30 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:30.435 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:30.435 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:30.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:30.563 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:36:30.566 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:30.566 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:30.674 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:30.808 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:30.940 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:31.076 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:31.207 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:31.229 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:31.229 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:31.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:31 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:31.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:31 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:31.229 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:31 smithi086 ceph-mon[29229]: osdmap e822: 12 total, 3 up, 11 in 2024-08-06T18:36:31.338 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:31.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:31 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:31.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:31 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:31.340 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:31 smithi117 ceph-mon[32835]: osdmap e822: 12 total, 3 up, 11 in 2024-08-06T18:36:31.363 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:31 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:36:31.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:31 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1906980416' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:31.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:31 smithi107 ceph-mon[32381]: osdmap e822: 12 total, 3 up, 11 in 2024-08-06T18:36:31.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:31.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:31.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:31.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:31.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:31.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:31.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:31.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:31.873 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:31.919 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:31.919 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:31.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:31.919 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:32.009 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:32.110 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:32.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:32.152 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:32.152 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:32.152 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:32.170 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:32.170 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:32.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:32 smithi086 ceph-mon[29229]: pgmap v2413: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:32.171 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:32 smithi086 ceph-mon[29229]: osdmap e823: 12 total, 3 up, 11 in 2024-08-06T18:36:32.171 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:32.171 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:32.171 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:32.189 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:32.190 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:32.190 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:32.190 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:32.190 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:32.190 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:32.191 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:32.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:32 smithi117 ceph-mon[32835]: pgmap v2413: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:32.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:32 smithi117 ceph-mon[32835]: osdmap e823: 12 total, 3 up, 11 in 2024-08-06T18:36:32.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:32.432 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:32 smithi107 ceph-mon[32381]: pgmap v2413: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:32.433 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:32 smithi107 ceph-mon[32381]: osdmap e823: 12 total, 3 up, 11 in 2024-08-06T18:36:32.433 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:32.433 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:32.542 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:32.640 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:32.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:36:32.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:32.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:32.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:32.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:36:32.814 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:32.905 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:32.906 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:32.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:32.951 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:33.074 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:33.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:33.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3940164652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:33.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:33 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:33.312 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:33.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3940164652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:33.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.312 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:33 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:33.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3940164652' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:33.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.362 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:33 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:33.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:33.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:33.733 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:33.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:33.994 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:34.123 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:34.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:34.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:34 smithi086 ceph-mon[29229]: pgmap v2415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:34.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:34 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:36:34.220 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:34 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:36:34.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:34.289 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:34 smithi107 ceph-mon[32381]: pgmap v2415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:34.290 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:34 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:36:34.290 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:34 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:36:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:34 smithi117 ceph-mon[32835]: pgmap v2415: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:34 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:36:34.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:34 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:36:34.399 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:34.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:34.565 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:34.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:34.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:34.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:36:34.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:35.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:35.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:35.079 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:35.202 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:35.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:35.362 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:35.363 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:35.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:35.606 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:35.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:35.877 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:36.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:36.040 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:36:36.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:36.155 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:36.155 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:36.156 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:36 smithi086 ceph-mon[29229]: pgmap v2416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:36.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:36.303 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:36 smithi117 ceph-mon[32835]: pgmap v2416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:36.413 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:36.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:36 smithi107 ceph-mon[32381]: pgmap v2416: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:36.548 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:36.684 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:36.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:36.841 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:36.841 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:36.951 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:37.023 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:37.084 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:37.106 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:37.106 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:37.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:37.216 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:37.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:37.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:37.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:37.484 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:37.614 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:37.751 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:37.773 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:37.773 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:37.881 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:38.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:38.036 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:36:38.037 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:38.037 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:38.154 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:38.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:38.310 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:38 smithi107 ceph-mon[32381]: pgmap v2417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:38.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:38.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:38 smithi086 ceph-mon[29229]: pgmap v2417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:38.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:38 smithi117 ceph-mon[32835]: pgmap v2417: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:36:38.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:38.556 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:38.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:38.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:38.963 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:39.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:39.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:39 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:39.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:39.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:39 smithi086 ceph-mon[29229]: osdmap e824: 12 total, 3 up, 11 in 2024-08-06T18:36:39.181 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:39 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:39 smithi107 ceph-mon[32381]: osdmap e824: 12 total, 3 up, 11 in 2024-08-06T18:36:39.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:39.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:39.233 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:39.365 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:39.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:39 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:39.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1235827991' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:36:39.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:39 smithi117 ceph-mon[32835]: osdmap e824: 12 total, 3 up, 11 in 2024-08-06T18:36:39.425 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:39.425 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:39.425 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:39.442 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:39.443 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:39.461 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:39.462 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:39.463 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:39.500 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:39.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:39.768 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:39.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:40.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:40.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:40.303 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:40.329 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:40 smithi117 ceph-mon[32835]: pgmap v2419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:40.329 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:40 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/144094367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:40.329 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.329 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:40 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:40 smithi086 ceph-mon[29229]: pgmap v2419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:40.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:40 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/144094367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:40.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:40 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.439 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:40.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:40 smithi107 ceph-mon[32381]: pgmap v2419: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:40.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:40 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/144094367' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:40.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:40 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:40.574 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:40.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:40.846 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:40.881 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:36:40.897 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:40.898 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:40.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:41.115 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:41.251 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:41.277 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:41.352 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:41.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:41.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:41.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:41.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:41.657 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:41.793 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:41.890 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:41.894 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:41.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:42.063 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:42.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:42 smithi107 ceph-mon[32381]: pgmap v2420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:42.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:42 smithi107 ceph-mon[32381]: osdmap e825: 12 total, 3 up, 11 in 2024-08-06T18:36:42.221 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:42 smithi117 ceph-mon[32835]: pgmap v2420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:42.222 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:42 smithi117 ceph-mon[32835]: osdmap e825: 12 total, 3 up, 11 in 2024-08-06T18:36:42.331 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:42.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:42 smithi086 ceph-mon[29229]: pgmap v2420: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:42.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:42 smithi086 ceph-mon[29229]: osdmap e825: 12 total, 3 up, 11 in 2024-08-06T18:36:42.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:42.334 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:42.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:42.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:42.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:42.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:42.734 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:42.864 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:42.865 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:36:42.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:42.899 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:36:43.006 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:43.031 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:43.032 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:43.141 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:43.277 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:43.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:43.408 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:43.544 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:43.674 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:43.804 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:43.934 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:43.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:43.943 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:44.070 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:44.110 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:44 smithi086 ceph-mon[29229]: pgmap v2422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:44.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:44 smithi107 ceph-mon[32381]: pgmap v2422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:44.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:44.363 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:44 smithi117 ceph-mon[32835]: pgmap v2422: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:44.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:44.608 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:44.744 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:44.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:45.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:45.144 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:45.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:45.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:45.545 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:45.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:45.813 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:45.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:46.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:46.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:46 smithi086 ceph-mon[29229]: pgmap v2423: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:46.116 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:45 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:46 smithi117 ceph-mon[32835]: pgmap v2423: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:46.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:46.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:46.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:46 smithi107 ceph-mon[32381]: pgmap v2423: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:46.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:46.618 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:46.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:46.752 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:46.754 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:46.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:47.021 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:47.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:47.288 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:47.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:47 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:36:47.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:47 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:36:47.289 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:47 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:36:47.289 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:47.290 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:47.314 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:47.314 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:47.314 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:47.331 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:47.332 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:47.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:47.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:47.332 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:47.349 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:47.349 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:47.350 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:47 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:36:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:47 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:36:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:47 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:36:47.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:47 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:36:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:47 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:36:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:47 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:36:47.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:47.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:47.550 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:47.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:47.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:47.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:47.576 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:47.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:47.811 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:47.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:47.986 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:36:48.073 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:48.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:48.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: pgmap v2424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: osdmap e826: 12 total, 3 up, 11 in 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/445697790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.116 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:36:48.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:36:48.117 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.128 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:36:48.205 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: pgmap v2424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: osdmap e826: 12 total, 3 up, 11 in 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.227 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/445697790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:36:48.228 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.335 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: pgmap v2424: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: osdmap e826: 12 total, 3 up, 11 in 2024-08-06T18:36:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.364 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/445697790' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:36:48.365 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:36:48.464 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:48.595 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:48.725 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:48.742 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:48.742 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:48.855 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:48.992 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:49.123 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:49.166 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:36:49.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:36:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:49 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:49.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:49 smithi107 ceph-mon[32381]: osdmap e827: 12 total, 3 up, 11 in 2024-08-06T18:36:49.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:49.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:48 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:49.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:49.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:49.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:49 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:49.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:49 smithi086 ceph-mon[29229]: osdmap e827: 12 total, 3 up, 11 in 2024-08-06T18:36:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:49 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/692951500' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:36:49.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:49 smithi117 ceph-mon[32835]: osdmap e827: 12 total, 3 up, 11 in 2024-08-06T18:36:49.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:49.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:49.652 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:49.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:36:49.707 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:36:49.783 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:49.790 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:49.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:49.935 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:49.936 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:50.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:50.186 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:50.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:50 smithi086 ceph-mon[29229]: pgmap v2427: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:50.275 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:50.276 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:50.321 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:50.453 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:50.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:50 smithi107 ceph-mon[32381]: pgmap v2427: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:50.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:50.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:50 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:50.479 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:50 smithi117 ceph-mon[32835]: pgmap v2427: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:50.598 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:50.729 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:50.743 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:36:50.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:50.992 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:51.125 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:51.256 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:51.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:51.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:51.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:51.261 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:51 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:51.391 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:51.471 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:51.495 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:36:51.495 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:36:51.523 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:51.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:51.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:36:51.785 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:51.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:52.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:52.182 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:52.203 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:52 smithi117 ceph-mon[32835]: pgmap v2428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:52.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:52.336 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:52 smithi107 ceph-mon[32381]: pgmap v2428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:52.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:52 smithi086 ceph-mon[29229]: pgmap v2428: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:52.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:52.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:52.711 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:52.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:52.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:52.734 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:52.734 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:52 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:52.842 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:52.935 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:36:52.975 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:53.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:53.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:53.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:53.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:53.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:53.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:53.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:53.769 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:53.772 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:53.956 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:54.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:54.031 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:54.065 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:54.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:54.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:54.292 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:54 smithi086 ceph-mon[29229]: pgmap v2429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:54.362 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:54.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:54 smithi117 ceph-mon[32835]: pgmap v2429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:54.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:54 smithi107 ceph-mon[32381]: pgmap v2429: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:54.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:54.543 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:54.543 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:54.631 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:54.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:54.797 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:54.900 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:55.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:55.057 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:55.145 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:36:55.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:36:55.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:55 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:36:55.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:55 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:36:55.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:55 smithi107 ceph-mon[32381]: osdmap e828: 12 total, 3 up, 11 in 2024-08-06T18:36:55.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:55 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:55.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:55 smithi107 ceph-mon[32381]: pgmap v2431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:55.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:55.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:55 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:36:55.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:55 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:36:55.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:55 smithi086 ceph-mon[29229]: osdmap e828: 12 total, 3 up, 11 in 2024-08-06T18:36:55.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:55 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:55.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:55 smithi086 ceph-mon[29229]: pgmap v2431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:55.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:55.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:55.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:55 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:36:55.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:55 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:36:55.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:55 smithi117 ceph-mon[32835]: osdmap e828: 12 total, 3 up, 11 in 2024-08-06T18:36:55.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:55 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:36:55.600 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:55 smithi117 ceph-mon[32835]: pgmap v2431: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:55.709 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:55.844 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:55.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:55.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:55 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:55.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:56.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:56.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:36:56.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:36:56.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:36:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:36:56.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:56 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:56.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:56.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:56 smithi086 ceph-mon[29229]: osdmap e829: 12 total, 3 up, 11 in 2024-08-06T18:36:56.375 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:56.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:56 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:56.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:56.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:56 smithi107 ceph-mon[32381]: osdmap e829: 12 total, 3 up, 11 in 2024-08-06T18:36:56.508 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:56.620 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:36:56.620 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:36:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:56 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:36:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3455825497' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:36:56.623 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:56 smithi117 ceph-mon[32835]: osdmap e829: 12 total, 3 up, 11 in 2024-08-06T18:36:56.639 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:56.661 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:56.661 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:56 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:56.770 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:56.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:36:57.036 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:57.166 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:57.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:57.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:57.392 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:36:57.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:57.575 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:36:57.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:57.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:57.844 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:57.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:58.096 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:57 smithi086 ceph-mon[29229]: pgmap v2433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:58.105 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:36:58.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:57 smithi117 ceph-mon[32835]: pgmap v2433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:58.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:57 smithi107 ceph-mon[32381]: pgmap v2433: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:58.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:58.334 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:36:58.338 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:36:58.338 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:36:58.338 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:36:58.360 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:36:58.360 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:36:58.360 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:36:58.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:36:58.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:36:58.361 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:36:58.361 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:36:58.367 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:36:58.378 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:36:58.379 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:36:58.379 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:36:58.379 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:36:58.379 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:36:58.379 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:36:58.380 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:36:58.445 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:36:58.502 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:58.636 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:36:58.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:36:58.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:59.035 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:59.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3460444976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:59.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:58 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:59.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:59.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3460444976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:59.192 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:58 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:59.302 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:36:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3460444976' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:36:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:58 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:59.432 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:36:59.458 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:59.458 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:59.567 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:36:59.700 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:59.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:36:59.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:36:59 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:36:59.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:36:59.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:59.948 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:59 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:36:59.949 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:36:59 smithi086 ceph-mon[29229]: pgmap v2434: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:36:59.962 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: "size": 20, 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 600, 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [ 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:673,from:2,reservation_nonce:27)", 2024-08-06T18:37:00.074 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 550.62079648700001, 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000145129, 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.075 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430362+0000", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430364+0000", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.513e-06 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430371+0000", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0190000000000001e-06 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.076 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430385+0000", 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4362000000000001e-05 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430396+0000", 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0229e-05 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430466+0000", 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.0858000000000005e-05 2024-08-06T18:37:00.077 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430481+0000", 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4596999999999999e-05 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430508+0000", 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.6551e-05 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.078 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:673/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 550.62018874399996, 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000111407, 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.079 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430970+0000", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430971+0000", 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6599999999999996e-07 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.080 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430975+0000", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.2889999999999999e-06 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430986+0000", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0729e-05 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.430992+0000", 2024-08-06T18:37:00.081 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.694e-06 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431040+0000", 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8418999999999998e-05 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431050+0000", 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.8549999999999997e-06 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.082 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431082+0000", 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.1754999999999997e-05 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:673,from:2,reservation_nonce:0)", 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 550.61957551600005, 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00012740599999999999, 2024-08-06T18:37:00.083 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431583+0000", 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.084 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431584+0000", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.36e-07 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431586+0000", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0080000000000001e-06 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431590+0000", 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0090000000000001e-06 2024-08-06T18:37:00.085 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431596+0000", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2759999999999997e-06 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431681+0000", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4974000000000003e-05 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431696+0000", 2024-08-06T18:37:00.086 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4654000000000001e-05 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:27:49.431711+0000", 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5248999999999999e-05 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:697,from:2,reservation_nonce:28)", 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 474.827708186, 2024-08-06T18:37:00.087 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00014356300000000001, 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223451+0000", 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.088 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223452+0000", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3510000000000001e-06 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223460+0000", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9049999999999997e-06 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223475+0000", 2024-08-06T18:37:00.089 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5021999999999999e-05 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223483+0000", 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.0350000000000001e-06 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223540+0000", 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.7235999999999999e-05 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.090 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223561+0000", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0309000000000002e-05 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.223594+0000", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.3704999999999997e-05 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:697,from:2,reservation_nonce:0)", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:37:00.091 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 474.82653481099999, 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000110475, 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.092 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224624+0000", 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.77e-07 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224627+0000", 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.424e-06 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.093 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224633+0000", 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.2169999999999999e-06 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224638+0000", 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0010000000000001e-06 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224712+0000", 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.4124999999999999e-05 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.094 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224723+0000", 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0895e-05 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:05.224735+0000", 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1436e-05 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,QREQUEST,e:699,from:6,reservation_nonce:30)", 2024-08-06T18:37:00.095 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 468.75808064900002, 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00022859099999999999, 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293078+0000", 2024-08-06T18:37:00.096 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293081+0000", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.4099999999999998e-06 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293094+0000", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.2809999999999999e-05 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293125+0000", 2024-08-06T18:37:00.097 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.0998999999999999e-05 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293139+0000", 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.4667e-05 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293256+0000", 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000116816 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.098 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293272+0000", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5574e-05 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.293307+0000", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.5315000000000003e-05 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.19,RELEASE,e:699,from:6,reservation_nonce:0)", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 468.75647199500003, 2024-08-06T18:37:00.099 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00016018, 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.100 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294687+0000", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.7e-07 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294707+0000", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.9466e-05 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294716+0000", 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.6660000000000003e-06 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.101 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294722+0000", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.6489999999999996e-06 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294825+0000", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000102699 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294838+0000", 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3339999999999999e-05 2024-08-06T18:37:00.102 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:29:11.294847+0000", 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.9900000000000003e-06 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:764,from:2,reservation_nonce:31)", 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:37:00.103 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 234.211279684, 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000136048, 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.104 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839879+0000", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839881+0000", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.0090000000000002e-06 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839889+0000", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2099999999999993e-06 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.105 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839905+0000", 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5970000000000001e-05 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839914+0000", 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.1839999999999999e-06 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839972+0000", 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.8715999999999998e-05 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.106 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.839991+0000", 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8578999999999999e-05 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840015+0000", 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.438e-05 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.107 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:764/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 234.21072515500001, 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000112552, 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.108 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840434+0000", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2399999999999998e-07 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840438+0000", 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.0589999999999996e-06 2024-08-06T18:37:00.109 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840450+0000", 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.1161e-05 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840455+0000", 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.3539999999999999e-06 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.110 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840503+0000", 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.8044000000000003e-05 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840513+0000", 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0468999999999999e-05 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.840546+0000", 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.2941000000000002e-05 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.111 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:764,from:2,reservation_nonce:0)", 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 234.21005979500001, 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000115126, 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.112 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841099+0000", 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.5400000000000002e-07 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.113 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841101+0000", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.7859999999999999e-06 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841105+0000", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.8399999999999997e-06 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841110+0000", 2024-08-06T18:37:00.114 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.7550000000000002e-06 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841194+0000", 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.4648999999999995e-05 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841205+0000", 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0713000000000001e-05 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.115 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:33:05.841214+0000", 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.1290000000000002e-06 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,QREQUEST,e:812,from:2,reservation_nonce:33)", 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:35:58.687679+0000", 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.363480101999997, 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000155344, 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.116 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687679+0000", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687679+0000", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687680+0000", 2024-08-06T18:37:00.117 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5689999999999999e-06 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687689+0000", 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.2479999999999996e-06 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687704+0000", 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5553000000000002e-05 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.118 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687713+0000", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 8.789e-06 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687779+0000", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.6445000000000005e-05 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687798+0000", 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.8621e-05 2024-08-06T18:37:00.119 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.687834+0000", 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6118999999999998e-05 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "replica_scrub(pg: 2.1,from:0'0,to:0'0,epoch:812/121,start:2:80000000::::0,end:MAX,chunky:1,deep:0,version:9,allow_preemption:1,priority=5)", 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:35:58.688277+0000", 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.362881448000003, 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.00010242, 2024-08-06T18:37:00.120 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688277+0000", 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688277+0000", 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.121 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688278+0000", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.8199999999999999e-07 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688283+0000", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 4.4279999999999998e-06 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688293+0000", 2024-08-06T18:37:00.122 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.0132999999999999e-05 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688301+0000", 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 7.9389999999999997e-06 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.123 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688351+0000", 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.0108999999999997e-05 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688357+0000", 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.5910000000000001e-06 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688380+0000", 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.2538e-05 2024-08-06T18:37:00.124 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "description": "MOSDScrubReserve(2.1,RELEASE,e:812,from:2,reservation_nonce:0)", 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "initiated_at": "2024-08-06T18:35:58.688967+0000", 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "age": 61.362191471000003, 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0.000134167, 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "continuous": false, 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "type_data": { 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "flag_point": "started", 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "events": [ 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "initiated", 2024-08-06T18:37:00.125 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688967+0000", 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "header_read", 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688967+0000", 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 0 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "throttled", 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688968+0000", 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 3.6100000000000002e-07 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.126 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "all_read", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688970+0000", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 2.3599999999999999e-06 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "dispatched", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688977+0000", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 6.567e-06 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "queued_for_pg", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.688982+0000", 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 5.2700000000000004e-06 2024-08-06T18:37:00.127 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "reached_pg", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.689073+0000", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 9.0531000000000004e-05 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "started", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.689088+0000", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.5497000000000001e-05 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: }, 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: { 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "event": "done", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "time": "2024-08-06T18:35:58.689102+0000", 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: "duration": 1.3580999999999999e-05 2024-08-06T18:37:00.128 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.129 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.129 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.129 INFO:teuthology.orchestra.run.smithi086.stdout: } 2024-08-06T18:37:00.129 INFO:teuthology.orchestra.run.smithi086.stdout: ] 2024-08-06T18:37:00.129 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:37:00.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:59 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:00.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:36:59 smithi107 ceph-mon[32381]: pgmap v2434: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:00.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:00.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:00.288 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:36:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:59 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:36:59 smithi117 ceph-mon[32835]: pgmap v2434: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:00.398 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:00.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:00.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:00.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:00 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:00.669 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:00.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:00.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:00 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:00.805 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:00.936 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:00.948 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:00.949 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:01.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:01.207 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:01.341 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:01.450 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_ops_in_flight 2024-08-06T18:37:01.451 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:01.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:01.609 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:01.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:01.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:01.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:01.878 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:01.931 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:01.932 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:02.012 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:02.034 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:01 smithi086 ceph-mon[29229]: pgmap v2435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:02.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:02.034 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:02.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:01 smithi117 ceph-mon[32835]: pgmap v2435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:02.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:02.170 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:02.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:01 smithi107 ceph-mon[32381]: pgmap v2435: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:02.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:02.292 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:02.292 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:02.411 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:02.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:02.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:02.811 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:02.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:02.949 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:37:02.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:02.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:03.080 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:03.209 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:03.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:03.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:03.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:03.480 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:03.614 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:03.750 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:03.882 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:03.885 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:04.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:04.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:04.179 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:03 smithi107 ceph-mon[32381]: pgmap v2436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:04.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:04.179 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:04.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:03 smithi086 ceph-mon[29229]: pgmap v2436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:04.289 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:04.314 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:03 smithi117 ceph-mon[32835]: pgmap v2436: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:04.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:04.485 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:04.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_blocked_ops 2024-08-06T18:37:04.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:04.692 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:04.823 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:04.959 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:05.012 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:05.016 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:05.095 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:05.231 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:05.363 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:05.497 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:05.633 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:05.763 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:05.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:06.033 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:06.059 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:05 smithi107 ceph-mon[32381]: pgmap v2437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:06.060 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:06.060 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:05 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:06.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:05 smithi086 ceph-mon[29229]: pgmap v2437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:06.169 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:06.301 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:06.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:05 smithi117 ceph-mon[32835]: pgmap v2437: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:06.437 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:06.572 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:06.597 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:06.597 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:06.706 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:06.838 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:06.866 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:06.866 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:06 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:06.947 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:37:06.976 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:06.984 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:06 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:06.984 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:06.984 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:07.106 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:07.132 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:06 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:07.242 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:07.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:06 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:07.376 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:07.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:07.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:07.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:07.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:07.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:07.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:07.776 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:07.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:08.041 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:07 smithi117 ceph-mon[32835]: pgmap v2438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:07 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:08.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:07 smithi117 ceph-mon[32835]: osdmap e830: 12 total, 3 up, 11 in 2024-08-06T18:37:08.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:07 smithi107 ceph-mon[32381]: pgmap v2438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:07 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:08.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:07 smithi107 ceph-mon[32381]: osdmap e830: 12 total, 3 up, 11 in 2024-08-06T18:37:08.249 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:08.249 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:08.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:07 smithi086 ceph-mon[29229]: pgmap v2438: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:08.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:07 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:08.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2210293952' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:08.249 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:07 smithi086 ceph-mon[29229]: osdmap e830: 12 total, 3 up, 11 in 2024-08-06T18:37:08.258 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:08.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:08.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:08.582 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:08.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:08.713 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:08 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:08.713 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:08.848 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:08.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:09.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:09.131 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:09.131 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:09.240 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:09.370 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:09.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:09.492 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:09 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:09.505 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:09.636 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:09.766 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:09.816 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:09.816 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:09.816 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:09.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:09.818 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:09.902 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:10.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:10.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:09 smithi086 ceph-mon[29229]: pgmap v2440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:10.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:09 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:37:10.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:09 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:37:10.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/430255957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:10.054 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:09 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:09 smithi117 ceph-mon[32835]: pgmap v2440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:09 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:37:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:09 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:37:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/430255957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:10.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:09 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:10.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:09 smithi107 ceph-mon[32381]: pgmap v2440: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:09 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:37:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:09 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:37:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/430255957' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:10.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:09 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:10.298 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:10.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:10.365 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:10.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:10.522 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:10.523 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:10.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:10.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:10.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:10.962 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:10.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.1 dump_historic_ops 2024-08-06T18:37:10.963 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:11.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:11.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.120 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:10 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:37:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:10 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.213 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:10 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:11.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:11.213 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:11.214 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:11.229 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:11.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:11.492 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:11.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:11.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:11.894 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:12.029 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:12.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:11 smithi107 ceph-mon[32381]: pgmap v2441: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:12.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:11 smithi107 ceph-mon[32381]: osdmap e831: 12 total, 3 up, 11 in 2024-08-06T18:37:12.296 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:12.318 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:11 smithi086 ceph-mon[29229]: pgmap v2441: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:12.318 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:11 smithi086 ceph-mon[29229]: osdmap e831: 12 total, 3 up, 11 in 2024-08-06T18:37:12.318 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:12.318 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:11 smithi117 ceph-mon[32835]: pgmap v2441: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:12.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:11 smithi117 ceph-mon[32835]: osdmap e831: 12 total, 3 up, 11 in 2024-08-06T18:37:12.418 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:12.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:12.524 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:37:12.562 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:12.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:12.583 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:12.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:12.834 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:12.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:12.969 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:13.103 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:13.113 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:13.238 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:13.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:13.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:13.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:13.777 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:13.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:14.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:14.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:13 smithi117 ceph-mon[32835]: pgmap v2443: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:14.175 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:13 smithi086 ceph-mon[29229]: pgmap v2443: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:14.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:14.208 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:14.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_ops_in_flight 2024-08-06T18:37:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:13 smithi107 ceph-mon[32381]: pgmap v2443: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:14.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:14.320 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:14.451 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:14.586 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:14.722 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:14.837 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:14.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:14.993 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:15.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:15.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:15.129 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:15.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:15.400 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:15.536 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:15.645 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:15.669 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:15.691 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:15.691 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:15.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:15.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:15.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:15.964 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:15.964 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:15.964 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:15 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:16.066 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:16.196 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:16.222 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:15 smithi117 ceph-mon[32835]: pgmap v2444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:16.255 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:15 smithi086 ceph-mon[29229]: pgmap v2444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:16.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:16.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:15 smithi107 ceph-mon[32381]: pgmap v2444: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:16.468 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:16.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:16.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:16.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:16.765 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:16 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:16.875 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:17.007 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:17.093 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:17.093 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:16 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:17.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:17.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:17.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:17.545 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:17.569 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:17.569 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:17.570 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:17.570 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:17.679 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:17.810 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:17.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:17.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:17 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:17.945 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:17.967 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:37:18.081 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:18.108 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:17 smithi107 ceph-mon[32381]: pgmap v2445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:18.108 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:18.218 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:18.241 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:17 smithi086 ceph-mon[29229]: pgmap v2445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:18.241 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:18.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:17 smithi117 ceph-mon[32835]: pgmap v2445: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:18.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:18.486 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:18.617 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:18.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:18.890 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:19.018 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:37:19.018 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:37:19.025 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:19.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:18 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:19.050 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:19.051 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:18 smithi117 ceph-mon[32835]: osdmap e832: 12 total, 3 up, 11 in 2024-08-06T18:37:19.160 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:19.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:19.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:18 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:19.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:18 smithi107 ceph-mon[32381]: osdmap e832: 12 total, 3 up, 11 in 2024-08-06T18:37:19.294 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:19.315 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:18 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:19.315 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/105253638' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:19.315 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:18 smithi086 ceph-mon[29229]: osdmap e832: 12 total, 3 up, 11 in 2024-08-06T18:37:19.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:19.555 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:19.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:19.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:19.691 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:19.718 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_blocked_ops 2024-08-06T18:37:19.826 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:19.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:20.097 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:20.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:19 smithi117 ceph-mon[32835]: pgmap v2447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:20.194 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:20.194 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:20.194 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:20.211 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:20.211 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:20.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:20.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:20.212 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:20.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:20.229 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:20.230 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:19 smithi086 ceph-mon[29229]: pgmap v2447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:20.231 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:20.231 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:20.231 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:20.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:20.232 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:20.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:20.233 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:20.360 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:20.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:19 smithi107 ceph-mon[32381]: pgmap v2447: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:20.492 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:20.513 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:20.513 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:20.622 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:20.730 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:37:20.751 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:20.882 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:21.014 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:21.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:20 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3191480100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:21.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.036 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:20 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.144 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:21.167 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:21.167 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:20 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3191480100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.167 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:20 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.276 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:21.406 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:21.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:20 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3191480100' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:21.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:20 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:21.535 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:21.665 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:21.738 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:21.738 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:21.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:21.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:21.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:21.925 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:22.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:22.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:21 smithi086 ceph-mon[29229]: pgmap v2448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:22.180 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:21 smithi086 ceph-mon[29229]: osdmap e833: 12 total, 3 up, 11 in 2024-08-06T18:37:22.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:22.246 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:22.319 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:22.345 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:21 smithi107 ceph-mon[32381]: pgmap v2448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:22.345 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:21 smithi107 ceph-mon[32381]: osdmap e833: 12 total, 3 up, 11 in 2024-08-06T18:37:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:21 smithi117 ceph-mon[32835]: pgmap v2448: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:22.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:21 smithi117 ceph-mon[32835]: osdmap e833: 12 total, 3 up, 11 in 2024-08-06T18:37:22.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:22.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:22.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:22.726 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:22.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:22.735 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:22 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:22.835 INFO:teuthology.orchestra.run.smithi086.stdout:{ 2024-08-06T18:37:22.835 INFO:teuthology.orchestra.run.smithi086.stdout: "ops": [], 2024-08-06T18:37:22.835 INFO:teuthology.orchestra.run.smithi086.stdout: "complaint_time": 30, 2024-08-06T18:37:22.835 INFO:teuthology.orchestra.run.smithi086.stdout: "num_blocked_ops": 0 2024-08-06T18:37:22.835 INFO:teuthology.orchestra.run.smithi086.stdout:} 2024-08-06T18:37:22.860 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:22.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:23.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:23.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:23.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:23.530 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:23.638 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.2 dump_historic_ops 2024-08-06T18:37:23.666 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:23.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:37:23.812 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:23.887 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:23.887 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:23.887 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:23.888 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:23.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:24.079 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:24.210 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:24.236 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:23 smithi117 ceph-mon[32835]: pgmap v2450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:24.346 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:24.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:23 smithi086 ceph-mon[29229]: pgmap v2450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:24.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:24.360 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:23 smithi107 ceph-mon[32381]: pgmap v2450: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:24.478 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:24.614 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:24.732 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:24.745 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:24.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:25.012 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:25.146 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:25.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:24 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:25.281 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:25.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:25.551 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:25.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:25.573 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:25.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:25.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:25.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:25 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:25.791 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:25.838 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:25.974 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:26.003 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:37:26.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:25 smithi117 ceph-mon[32835]: pgmap v2451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:26.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:25 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:26.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:26.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:25 smithi086 ceph-mon[29229]: pgmap v2451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:26.265 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:25 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:26.266 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:26.266 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:26.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:26.401 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:25 smithi107 ceph-mon[32381]: pgmap v2451: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:26.402 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:25 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:26.511 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:26.533 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:26.533 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:26.642 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:26.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:26.786 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:26.786 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:26.913 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:27.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:27 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:27.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:27 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:27.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:27 smithi086 ceph-mon[29229]: osdmap e834: 12 total, 3 up, 11 in 2024-08-06T18:37:27.038 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:27 smithi086 ceph-mon[29229]: osdmap e835: 12 total, 3 up, 11 in 2024-08-06T18:37:27.052 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:27 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:27 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:27 smithi117 ceph-mon[32835]: osdmap e834: 12 total, 3 up, 11 in 2024-08-06T18:37:27.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:27 smithi117 ceph-mon[32835]: osdmap e835: 12 total, 3 up, 11 in 2024-08-06T18:37:27.184 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:27.206 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:27 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:27.207 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:27 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2300945830' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:27.207 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:27 smithi107 ceph-mon[32381]: osdmap e834: 12 total, 3 up, 11 in 2024-08-06T18:37:27.207 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:27 smithi107 ceph-mon[32381]: osdmap e835: 12 total, 3 up, 11 in 2024-08-06T18:37:27.315 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:27.447 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:27.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:27.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:27.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:27.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:27.616 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:27.714 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:27.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:27.986 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:28.119 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:28.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:28 smithi107 ceph-mon[32381]: pgmap v2453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:28.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:27 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:28.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:28.274 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:28 smithi117 ceph-mon[32835]: pgmap v2453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:28.360 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:28 smithi086 ceph-mon[29229]: pgmap v2453: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:28.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:28.360 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:28.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:28.519 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:28.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:28.747 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:28.748 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:28.748 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:28.763 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:28.764 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:28.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:28.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:28.764 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:28.781 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:28.782 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:28.783 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:28.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:28.815 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:37:28.919 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:28.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:28.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:29.050 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:29.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:29.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2436707750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:29.207 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:29 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:29.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:29.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2436707750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:29.341 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:29 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:29.451 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:29.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2436707750' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:29.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:29 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:29.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:29.581 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:29.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:29.722 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:29.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:29.940 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:37:29.941 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:37:29.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:30.112 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:30 smithi086 ceph-mon[29229]: pgmap v2455: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.248 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:30.274 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:30 smithi107 ceph-mon[32381]: pgmap v2455: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:30.274 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.274 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:30 smithi117 ceph-mon[32835]: pgmap v2455: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:30.383 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:30.514 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:30.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:30.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:30.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_ops_in_flight 2024-08-06T18:37:30.720 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:30.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:30.917 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:30.939 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:30.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:30.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:30.940 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:30 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:30.985 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:31.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:31.182 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:31.311 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:31.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:31.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:31.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:31.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:31.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:31.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:31.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:31 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:31.723 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:37:31.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:31.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:32.021 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:32.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:32.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:32 smithi117 ceph-mon[32835]: pgmap v2456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:32.239 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:32.261 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:32 smithi107 ceph-mon[32381]: pgmap v2456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:32.281 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:32 smithi086 ceph-mon[29229]: pgmap v2456: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:32.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:32.281 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:32 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:32.371 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:32.507 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:32.576 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:32.601 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:32.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:32.661 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:32.774 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:32.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:32.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:32.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:32 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:33.026 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:37:33.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:33.174 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:33.289 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:33 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:33.291 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_blocked_ops 2024-08-06T18:37:33.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:33.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:33 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:33.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:33 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:33.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:33.541 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:33.599 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:33.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:33.744 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:33.744 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:33.869 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:34.000 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:34.021 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:34.021 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:34.021 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:34.021 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:33 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:34.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:34.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: pgmap v2457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: osdmap e836: 12 total, 3 up, 11 in 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:37:34.287 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:34 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:37:34.310 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: pgmap v2457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:34.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:34.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:34.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: osdmap e836: 12 total, 3 up, 11 in 2024-08-06T18:37:34.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:37:34.311 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:34 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:37:34.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: pgmap v2457: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1768711154' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: osdmap e836: 12 total, 3 up, 11 in 2024-08-06T18:37:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:37:34.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:34 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:37:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:34.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:34.397 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:34.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:34.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:34.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:34.856 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:34.856 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:34.856 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:34.872 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:34.873 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:34.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:34.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:34.873 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:34.890 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:34.891 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:34.931 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:35.064 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:35.106 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:35 smithi117 ceph-mon[32835]: osdmap e837: 12 total, 3 up, 11 in 2024-08-06T18:37:35.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:35 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1941384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:35.124 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:35 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:35.197 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:35.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:35.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:35 smithi107 ceph-mon[32381]: osdmap e837: 12 total, 3 up, 11 in 2024-08-06T18:37:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:35 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1941384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:35.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:35 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:35.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:35.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:35.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:35.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:35 smithi086 ceph-mon[29229]: osdmap e837: 12 total, 3 up, 11 in 2024-08-06T18:37:35.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:35 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1941384' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:35.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:35 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:35.463 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:35.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:35.600 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:35.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:35.729 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:35.761 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:35.762 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.4 dump_historic_ops 2024-08-06T18:37:35.778 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:37:35.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:35.823 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:35.862 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:35.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:36.009 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:36.049 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:36 smithi086 ceph-mon[29229]: pgmap v2460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:36 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.127 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:36.257 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:36.287 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:36 smithi107 ceph-mon[32381]: pgmap v2460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:36.287 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.287 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:36 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:36 smithi117 ceph-mon[32835]: pgmap v2460: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:36 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:36.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:36.466 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:36.466 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:36.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:36.654 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:36.786 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:36.918 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:37.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:37.070 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:37.070 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:36 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:37.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:37.316 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:37.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:37.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:37.508 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:37.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:37.601 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:37:37.621 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:37.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:37.852 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:37.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:37.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:37.866 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:37.985 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:38.043 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:37:38.118 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:38.131 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:38 smithi086 ceph-mon[29229]: pgmap v2461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:38.131 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:38 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:38.131 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:38.132 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:38.240 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:38.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_ops_in_flight 2024-08-06T18:37:38.254 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:38.262 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:38 smithi107 ceph-mon[32381]: pgmap v2461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:38.262 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:38 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:38 smithi117 ceph-mon[32835]: pgmap v2461: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:38 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:38.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:38.492 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:38.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:38.660 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:38.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:38.815 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:38.815 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:38.924 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:39.055 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:39.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:39 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:39.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:39 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:39.078 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:39 smithi086 ceph-mon[29229]: osdmap e838: 12 total, 3 up, 11 in 2024-08-06T18:37:39.187 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:39.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:39.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:38 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:39 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:39 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:39.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:39 smithi107 ceph-mon[32381]: osdmap e838: 12 total, 3 up, 11 in 2024-08-06T18:37:39.317 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:39.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:39.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:39.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:39.332 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:39.343 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:39 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:37:39.343 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:39 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/565457033' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:39.343 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:39 smithi117 ceph-mon[32835]: osdmap e838: 12 total, 3 up, 11 in 2024-08-06T18:37:39.453 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:39.583 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:39.717 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:39.847 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:39.868 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:39.868 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:39.977 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:40.041 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:40.113 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:40.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:40 smithi107 ceph-mon[32381]: pgmap v2463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:40.236 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:40 smithi086 ceph-mon[29229]: pgmap v2463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:40.236 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:40.236 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:40.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:40.251 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:40.251 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:40.252 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:40.269 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:40.270 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:40.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:40.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:40.270 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:40.286 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:40.287 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:40.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:40.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:40.287 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:40.288 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:40.288 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:40.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:40 smithi117 ceph-mon[32835]: pgmap v2463: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:40.376 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:40.510 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:40.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:40.646 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:40.646 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_blocked_ops 2024-08-06T18:37:40.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:40.792 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:40.792 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:40.896 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:40.901 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:40.946 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:40.947 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:41.034 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:41.164 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:41.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:41.322 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:41 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3523970514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:41.322 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.322 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.322 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:41 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:37:41.337 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:41 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3523970514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:41.337 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.337 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.337 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:41 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.355 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:41 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3523970514' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:41.356 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.356 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.356 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:41 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:41.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:41.356 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:41.431 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:41.566 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:41.581 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:41.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:41.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:41.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:42.100 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:42.231 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:42.255 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:42 smithi107 ceph-mon[32381]: pgmap v2464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:42.256 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:42 smithi107 ceph-mon[32381]: osdmap e839: 12 total, 3 up, 11 in 2024-08-06T18:37:42.257 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:42 smithi117 ceph-mon[32835]: pgmap v2464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:42.258 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:42 smithi117 ceph-mon[32835]: osdmap e839: 12 total, 3 up, 11 in 2024-08-06T18:37:42.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:42 smithi086 ceph-mon[29229]: pgmap v2464: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:42.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:42 smithi086 ceph-mon[29229]: osdmap e839: 12 total, 3 up, 11 in 2024-08-06T18:37:42.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:42.469 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:42.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:42.625 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:42.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:42.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:42.948 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:37:42.967 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:42.968 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:42 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:43.019 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:43.021 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:43.021 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:43.049 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:43.049 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.5 dump_historic_ops 2024-08-06T18:37:43.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:43.282 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:43.301 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:43.414 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:43.439 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:43.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:43.560 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:43.561 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:43.681 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:43.816 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:43.947 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:44.061 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:37:44.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:44.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:44 smithi086 ceph-mon[29229]: pgmap v2466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:44.092 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:44 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:44.216 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:44.347 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:44.369 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:44 smithi107 ceph-mon[32381]: pgmap v2466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:44.369 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:44 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:44 smithi117 ceph-mon[32835]: pgmap v2466: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:44 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:37:44.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:44.608 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:44.739 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:44.847 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:44.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:45.004 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:45.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:45.187 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:45.187 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:45.187 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:45 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:45.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:45 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:45.188 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:45 smithi086 ceph-mon[29229]: osdmap e840: 12 total, 3 up, 11 in 2024-08-06T18:37:45.274 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:45.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:45 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:45.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:45 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:45.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:45 smithi117 ceph-mon[32835]: osdmap e840: 12 total, 3 up, 11 in 2024-08-06T18:37:45.407 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:45.429 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:45 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:45.430 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:45 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2874565921' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:37:45.430 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:45 smithi107 ceph-mon[32381]: osdmap e840: 12 total, 3 up, 11 in 2024-08-06T18:37:45.492 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:45.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_ops_in_flight 2024-08-06T18:37:45.538 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:45.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:45.738 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:45.809 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:45.815 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:45.815 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:45.815 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:45.816 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:45.817 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:45.943 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:46.076 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:46 smithi117 ceph-mon[32835]: pgmap v2468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4063226403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:46.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:46 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:46.153 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:46.153 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:46.153 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:46 smithi107 ceph-mon[32381]: pgmap v2468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:46.153 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4063226403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:46.153 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:46 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:46.208 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:46.340 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:46.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:46 smithi086 ceph-mon[29229]: pgmap v2468: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:46.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4063226403' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:46.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:46 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:46.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:46.517 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:46.604 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:46.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:46.831 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:37:46.867 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:46.998 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:47.078 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:47.079 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:47.080 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:47.134 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:47.265 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2024-08-06T18:37:47.292 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:37:47.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:47 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.340 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:47 smithi086 ceph-mon[29229]: osdmap e841: 12 total, 3 up, 11 in 2024-08-06T18:37:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:47 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:47 smithi117 ceph-mon[32835]: osdmap e841: 12 total, 3 up, 11 in 2024-08-06T18:37:47.395 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:47 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:47.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:47 smithi107 ceph-mon[32381]: osdmap e841: 12 total, 3 up, 11 in 2024-08-06T18:37:47.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:47.463 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:47.525 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:47.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:47.792 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:47.883 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_blocked_ops 2024-08-06T18:37:47.927 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:48.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:48.138 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:48.186 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:48 smithi107 ceph-mon[32381]: pgmap v2469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:48 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:37:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:48 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:37:48.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:48 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:37:48.318 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:48.347 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:48.347 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:48.347 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:48 smithi086 ceph-mon[29229]: pgmap v2469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:48.347 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:48 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:37:48.347 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:48 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:37:48.347 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:48 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:37:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:48 smithi117 ceph-mon[32835]: pgmap v2469: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:48 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:37:48.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:48 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:37:48.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:48 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:37:48.448 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:48.518 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:37:48.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:48.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:48.811 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:48.864 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:48.996 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:49.127 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:49.148 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: osdmap e842: 12 total, 3 up, 11 in 2024-08-06T18:37:49.149 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:37:49.149 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:37:49.149 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:37:49.149 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.149 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:49 smithi117 ceph-mon[32835]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.257 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:49.274 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: osdmap e842: 12 total, 3 up, 11 in 2024-08-06T18:37:49.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:37:49.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:37:49.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:37:49.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.275 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:49 smithi086 ceph-mon[29229]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.325 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:37:49.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: osdmap e842: 12 total, 3 up, 11 in 2024-08-06T18:37:49.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:37:49.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:37:49.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:37:49.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.374 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:49 smithi107 ceph-mon[32381]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:37:49.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:49.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:49.540 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:49.540 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:49.541 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:49.541 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:49.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:49.697 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2024-08-06T18:37:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2024-08-06T18:37:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "complaint_time": 30, 2024-08-06T18:37:49.697 INFO:teuthology.orchestra.run.smithi107.stdout: "num_blocked_ops": 0 2024-08-06T18:37:49.698 INFO:teuthology.orchestra.run.smithi107.stdout:} 2024-08-06T18:37:49.781 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:49.802 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:49.803 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:49.911 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:50.046 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:50.067 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:50.068 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:50.177 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:50 smithi107 ceph-mon[32381]: pgmap v2472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:50 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:50 smithi107 ceph-mon[32381]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:50.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:50 smithi107 ceph-mon[32381]: osdmap e843: 12 total, 3 up, 11 in 2024-08-06T18:37:50.313 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:50.320 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.6 dump_historic_ops 2024-08-06T18:37:50.334 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:50 smithi086 ceph-mon[29229]: pgmap v2472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:50.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:50 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:50.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:50 smithi086 ceph-mon[29229]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:50.335 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:50 smithi086 ceph-mon[29229]: osdmap e843: 12 total, 3 up, 11 in 2024-08-06T18:37:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:50 smithi117 ceph-mon[32835]: pgmap v2472: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:50 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:50 smithi117 ceph-mon[32835]: from='client.20064 172.21.15.86:0/2103747378' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:37:50.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:50 smithi117 ceph-mon[32835]: osdmap e843: 12 total, 3 up, 11 in 2024-08-06T18:37:50.443 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:50.516 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:50.517 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:50.517 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:50.532 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:50.533 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:50.533 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:50.534 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:50.534 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:50.550 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:50.551 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:50.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:50.583 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:50.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:50.595 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:50 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:50.704 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:50.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:50.965 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:51.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:51 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3844562821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:51.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:51 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:51.236 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:51.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:51 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3844562821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:51.258 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:51 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:51.367 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:51 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3844562821' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:51.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:51 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:51.448 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:51.449 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:51.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:51.635 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:51.765 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:51.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:52.045 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:52.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:52 smithi086 ceph-mon[29229]: pgmap v2474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:52.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:52 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout:ERROR: (22) Invalid argument 2024-08-06T18:37:52.159 INFO:teuthology.orchestra.run.smithi107.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:37:52.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:52.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:52 smithi117 ceph-mon[32835]: pgmap v2474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:52 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.444 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:52.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:52 smithi107 ceph-mon[32381]: pgmap v2474: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:52.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:52 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:52.573 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:52.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:52.762 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:52.762 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:52 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:52.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_ops_in_flight 2024-08-06T18:37:52.843 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:53.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:53.082 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:53.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:53.275 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:53.409 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:53.449 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:37:53.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:53.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:53.709 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:53.808 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:53.808 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:53 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:53.811 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:53.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:37:53.942 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:54.077 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:54.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:54.097 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:53 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:54.213 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:54.324 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:54 smithi086 ceph-mon[29229]: pgmap v2475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:54.324 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:54.325 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:54 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:54.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:54.371 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:54 smithi117 ceph-mon[32835]: pgmap v2475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:54.389 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:54 smithi107 ceph-mon[32381]: pgmap v2475: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:54.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:54.595 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:54.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:54.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:54.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:55.014 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:55.015 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:55.147 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:55.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_blocked_ops 2024-08-06T18:37:55.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:55.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:55.527 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:55.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:55.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:55.807 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:55.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:56.072 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:56.114 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:37:56.115 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:37:56.115 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:37:56.131 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:37:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:37:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:37:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:37:56.132 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:37:56.150 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:37:56.150 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:37:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:37:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:37:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:37:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:37:56.151 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:37:56.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:56.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:56.358 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:56 smithi086 ceph-mon[29229]: pgmap v2476: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:56.358 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:56 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3682642990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:56 smithi117 ceph-mon[32835]: pgmap v2476: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:56.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:56 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3682642990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:56.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:56 smithi107 ceph-mon[32381]: pgmap v2476: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:56.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:56 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3682642990' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:37:56.467 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:56.603 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:56.734 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:56.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:57.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:37:57.033 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:57.105 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:37:57.105 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:37:57 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:37:57.137 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:57.154 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.154 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.154 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:57 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.270 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:57 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.382 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:37:57.382 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:37:57.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:37:57.422 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.422 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.422 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:57 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:37:57.530 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:57.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:37:57.697 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:37:57.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.7 dump_historic_ops 2024-08-06T18:37:57.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:57.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:37:57.798 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:37:57 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:37:57.929 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:37:57.946 INFO:teuthology.orchestra.run.smithi107.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.b/config 2024-08-06T18:37:58.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:37:58.110 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:37:58.196 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:58.328 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:58.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:58 smithi086 ceph-mon[29229]: pgmap v2477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:58.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:58 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:58 smithi117 ceph-mon[32835]: pgmap v2477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:58.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:58 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:58.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:58 smithi107 ceph-mon[32381]: pgmap v2477: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:37:58.398 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:58 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:37:58.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:58.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:37:58.724 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:58.745 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:58.745 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:58.854 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:58.984 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:37:59.108 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:59.108 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:58 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:59.121 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:59.253 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:37:59.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:37:59.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:37:59 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:37:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:59 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:59 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:59.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:37:59 smithi086 ceph-mon[29229]: osdmap e844: 12 total, 3 up, 11 in 2024-08-06T18:37:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:59 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:59 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:59.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:37:59 smithi117 ceph-mon[32835]: osdmap e844: 12 total, 3 up, 11 in 2024-08-06T18:37:59.379 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:59 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:37:59.379 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:59 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/795586450' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:37:59.379 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:37:59 smithi107 ceph-mon[32381]: osdmap e844: 12 total, 3 up, 11 in 2024-08-06T18:37:59.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:37:59.384 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:37:59.463 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:37:59.514 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:37:59.634 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:37:59.644 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:37:59.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:37:59.913 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:00.048 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:00.091 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:00.092 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_ops_in_flight 2024-08-06T18:38:00.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:00.308 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:00.329 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:00 smithi086 ceph-mon[29229]: pgmap v2479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:00.345 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:00.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:00 smithi117 ceph-mon[32835]: pgmap v2479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:00.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:00.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:00 smithi107 ceph-mon[32381]: pgmap v2479: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:00.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:00.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:00.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:00.914 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:38:01.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:01.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:01.286 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:01.327 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:01.328 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:01.328 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:01.346 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:01.347 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:01.364 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:01.364 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:01.367 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:01.368 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:01.419 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:01.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:01.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:01 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:01.558 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:01.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:01.825 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:01.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:01.850 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:01 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:01.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:02.051 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:02.052 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:02.053 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:02.091 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:02.223 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:02.290 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: pgmap v2480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:02.290 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3219606195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:02.290 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.291 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.291 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.291 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:02 smithi117 ceph-mon[32835]: osdmap e845: 12 total, 3 up, 11 in 2024-08-06T18:38:02.379 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: pgmap v2480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3219606195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.388 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:02 smithi086 ceph-mon[29229]: osdmap e845: 12 total, 3 up, 11 in 2024-08-06T18:38:02.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: pgmap v2480: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:02.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3219606195' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:02.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:02.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:02 smithi107 ceph-mon[32381]: osdmap e845: 12 total, 3 up, 11 in 2024-08-06T18:38:02.512 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:02.590 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:02.591 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_blocked_ops 2024-08-06T18:38:02.648 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:02.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:02.853 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:02.912 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:02.934 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:02.934 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:02 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:02.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:02.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:02 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:03.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:03.173 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:03.194 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:03.194 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:03.304 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:03.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:03.452 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:03.452 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:03.573 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:03.706 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:03.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:03.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:03 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:03.836 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:03.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:03 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:03.967 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:04.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:38:04.099 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:04.144 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:38:04.230 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:04.307 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:04.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:04.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:04.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:04 smithi086 ceph-mon[29229]: pgmap v2482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:04.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:04 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:04.366 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:04 smithi117 ceph-mon[32835]: pgmap v2482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:04.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:04 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:04.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:04 smithi107 ceph-mon[32381]: pgmap v2482: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:04.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:04 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:04.501 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:04.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:04.766 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:04.789 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:04.789 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:04 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:04.851 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.8 dump_historic_ops 2024-08-06T18:38:04.897 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:04.953 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:05.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:05.100 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:05.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:05.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:05.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: 2.19 scrub starts 2024-08-06T18:38:05.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: 2.19 scrub ok 2024-08-06T18:38:05.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:05.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:05.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: osdmap e846: 12 total, 3 up, 11 in 2024-08-06T18:38:05.366 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:05 smithi086 ceph-mon[29229]: pgmap v2484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:05.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:05.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: 2.19 scrub starts 2024-08-06T18:38:05.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: 2.19 scrub ok 2024-08-06T18:38:05.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:05.448 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:05.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: osdmap e846: 12 total, 3 up, 11 in 2024-08-06T18:38:05.449 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:05 smithi117 ceph-mon[32835]: pgmap v2484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:05.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: 2.19 scrub starts 2024-08-06T18:38:05.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: 2.19 scrub ok 2024-08-06T18:38:05.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:05.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1353024549' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:05.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: osdmap e846: 12 total, 3 up, 11 in 2024-08-06T18:38:05.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:05 smithi107 ceph-mon[32381]: pgmap v2484: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:05.558 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:05.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:05.824 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:05.959 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:06.094 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:06.227 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:06.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:06.486 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:06.567 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:06.567 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:06.567 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:06.582 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:06.583 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:06.583 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:06.583 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:06.583 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:06.599 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:06.600 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:06.623 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:06.669 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:06.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:06.888 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:06.920 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:38:07.022 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:07.153 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:07.290 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:07.291 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:07.292 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_ops_in_flight 2024-08-06T18:38:07.313 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:07.313 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:06 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:07.313 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:07.313 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:07 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:07.340 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:07.340 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:07.392 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:07.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:07.578 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:07.597 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:07.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: pgmap v2485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:07.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/4017907994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:07 smithi086 ceph-mon[29229]: osdmap e847: 12 total, 3 up, 11 in 2024-08-06T18:38:07.733 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:07.796 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: pgmap v2485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:07.797 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/4017907994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:07.797 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.797 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.797 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.797 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:07 smithi117 ceph-mon[32835]: osdmap e847: 12 total, 3 up, 11 in 2024-08-06T18:38:07.864 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:07.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:07.963 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:07 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: pgmap v2485: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/4017907994' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:07.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:07 smithi107 ceph-mon[32381]: osdmap e847: 12 total, 3 up, 11 in 2024-08-06T18:38:07.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:08.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:08.261 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:08.393 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:08.416 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:08.416 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:08 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:08.524 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:08.657 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:08.793 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:08.921 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:08.925 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:09.057 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:09.192 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:09.323 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:09.342 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:38:09.452 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:09.477 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:09.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_blocked_ops 2024-08-06T18:38:09.545 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:38:09.588 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:09.615 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:09 smithi086 ceph-mon[29229]: pgmap v2487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:09.616 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:09 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:09.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:09.726 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:09.818 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:09 smithi117 ceph-mon[32835]: pgmap v2487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:09.818 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:09 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:09.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:09.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:09 smithi107 ceph-mon[32381]: pgmap v2487: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:09.876 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:09 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:09.985 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:10.117 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:10.217 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:10.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:10.388 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:10.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:10 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:10.519 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:10.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:10.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: purged_snaps scrub starts 2024-08-06T18:38:10.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: purged_snaps scrub ok 2024-08-06T18:38:10.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:10.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:10.671 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: osdmap e848: 12 total, 3 up, 11 in 2024-08-06T18:38:10.672 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:38:10.672 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:38:10.672 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:38:10.672 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:10 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:38:10.779 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:10.865 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: purged_snaps scrub starts 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: purged_snaps scrub ok 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: osdmap e848: 12 total, 3 up, 11 in 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:38:10.866 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:10 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:38:10.911 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:10.962 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: purged_snaps scrub starts 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: purged_snaps scrub ok 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1907771455' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: osdmap e848: 12 total, 3 up, 11 in 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "a"} : dispatch 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "b"} : dispatch 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "mon metadata", "id": "c"} : dispatch 2024-08-06T18:38:10.963 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:10 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config dump", "format": "json"} : dispatch 2024-08-06T18:38:11.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:11.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:11.220 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:11.312 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:11.350 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:11.351 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:10 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:11.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:11.579 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:11.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:11.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:11.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:11 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:11.858 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:11.883 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:11.884 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:11.974 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:11.974 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:11.975 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:11.986 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:11.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:11.992 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:11.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.9 dump_historic_ops 2024-08-06T18:38:11.994 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:12.005 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:12.006 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:12.006 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:12.006 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:12.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:12.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:12.007 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:12.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:12.238 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:12.248 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:38:12.264 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:12.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:12.269 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:11 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:12.394 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:12.525 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:12.662 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:12.792 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:12.797 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:12.831 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:12.926 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:12.940 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:12.940 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:12 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:13.062 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:13.197 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:12 smithi107 ceph-mon[32381]: osdmap e849: 12 total, 3 up, 11 in 2024-08-06T18:38:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:12 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1375655570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:12 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:13.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:13 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:13.218 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:12 smithi086 ceph-mon[29229]: osdmap e849: 12 total, 3 up, 11 in 2024-08-06T18:38:13.218 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:12 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1375655570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:13.219 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.219 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:12 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:12 smithi117 ceph-mon[32835]: osdmap e849: 12 total, 3 up, 11 in 2024-08-06T18:38:13.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:12 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1375655570' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:13.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.280 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:12 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:13.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:13.459 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:13.589 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:13.640 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:13.719 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:13.850 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:13.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:13.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:13.980 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:14.110 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:14.200 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:13 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:14.201 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:13 smithi117 ceph-mon[32835]: pgmap v2491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:13 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:14.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:13 smithi107 ceph-mon[32381]: pgmap v2491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:14.227 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:14.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_ops_in_flight 2024-08-06T18:38:14.239 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:13 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:14.240 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:13 smithi086 ceph-mon[29229]: pgmap v2491: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:14.240 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:14.240 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:13 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:14.241 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:14.377 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:14.473 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:14.507 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:14.507 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:14.507 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:14.638 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:14.775 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:14.799 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:38:14.813 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:14.814 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:14 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:14.907 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:15.043 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:15.095 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:15.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:15.199 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:15.199 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:14 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:15.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:38:15.230+0000 7fab6e59c640 0 --2- 172.21.15.86:0/353331827 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7fab70075100 0x7fab7019a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:38:15.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:15.439 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:15.570 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:15.702 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:15.835 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:15.880 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:15.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:15.990 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:15.990 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:15 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:16.092 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:38:16.099 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:16.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: pgmap v2492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.293 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:16 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:16.361 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: pgmap v2492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:38:16.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:16 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:16.462 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:16.463 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_blocked_ops 2024-08-06T18:38:16.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: pgmap v2492: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "config generate-minimal-conf"} : dispatch 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' cmd={"prefix": "auth get", "entity": "client.admin"} : dispatch 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='mgr.14148 172.21.15.86:0/204698182' entity='mgr.y' 2024-08-06T18:38:16.464 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:16 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:16.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:16.627 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:16.708 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:16.762 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:16.892 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:17.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:17.153 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:17.175 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:17 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:17.175 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:17 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:17.175 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:17 smithi117 ceph-mon[32835]: osdmap e850: 12 total, 3 up, 11 in 2024-08-06T18:38:17.176 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:17 smithi117 ceph-mon[32835]: osdmap e851: 12 total, 3 up, 11 in 2024-08-06T18:38:17.284 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:17.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:17 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:17.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:17 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:17.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:17 smithi086 ceph-mon[29229]: osdmap e850: 12 total, 3 up, 11 in 2024-08-06T18:38:17.320 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:17 smithi086 ceph-mon[29229]: osdmap e851: 12 total, 3 up, 11 in 2024-08-06T18:38:17.419 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:17.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:17 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:17.445 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:17 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/353331827' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:17.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:17 smithi107 ceph-mon[32381]: osdmap e850: 12 total, 3 up, 11 in 2024-08-06T18:38:17.446 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:17 smithi107 ceph-mon[32381]: osdmap e851: 12 total, 3 up, 11 in 2024-08-06T18:38:17.446 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:17.446 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:17 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:17.538 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:17.539 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:17.539 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:17.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:17.555 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:17.556 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:17.573 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:17.574 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:17.575 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:17.687 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:17.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:17.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:18.086 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:18 smithi086 ceph-mon[29229]: pgmap v2494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:18 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1230486279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:18.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:18 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:18.184 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:18.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:18.248 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:18.366 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:18.373 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:18.373 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:18 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:18.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:18 smithi107 ceph-mon[32381]: pgmap v2494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:18.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:18 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1230486279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:18.373 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:18 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:18.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:18 smithi117 ceph-mon[32835]: pgmap v2494: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:18.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:18 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1230486279' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:18.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:18 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:18.496 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:18.626 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:18.756 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:18.763 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:18.764 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.10 dump_historic_ops 2024-08-06T18:38:18.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:19.007 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:19.019 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:19.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:38:19.056 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:19.056 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:18 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:19.152 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:19 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.283 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:19.286 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:19.298 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.298 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:19 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:19 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:19.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:19.366 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:19.415 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:19.546 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:19.678 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:19.699 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:19.699 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:19.810 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:19.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:19.963 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:19.963 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:20.075 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:20.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:20.249 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:38:20.271 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:20 smithi086 ceph-mon[29229]: pgmap v2496: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:20.271 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:20.271 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:19 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:20.341 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:20.367 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:20 smithi107 ceph-mon[32381]: pgmap v2496: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:20.371 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:20 smithi117 ceph-mon[32835]: pgmap v2496: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:20.456 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:20.478 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:20.609 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:20.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:20.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:20 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:20.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:20.865 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:20 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:20.874 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:20.978 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:21.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_ops_in_flight 2024-08-06T18:38:21.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:21.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:21.267 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:21.271 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:21.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:21.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:21.668 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:21.804 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:21.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:21.865 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:21.938 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:22.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:22.103 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is unset 2024-08-06T18:38:22.200 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:22.337 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:22.363 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:22 smithi107 ceph-mon[32381]: pgmap v2497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:22.363 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:22 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:38:22.363 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:22.363 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:22 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:22.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:22 smithi086 ceph-mon[29229]: pgmap v2497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:22.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:22 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:38:22.365 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:22.366 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:21 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:22.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:22 smithi117 ceph-mon[32835]: pgmap v2497: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:22.368 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:22 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd={"prefix": "osd unset", "key": "noscrub"} : dispatch 2024-08-06T18:38:22.473 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:22.609 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:22.653 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:22.741 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:22.876 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:22.930 INFO:teuthology.orchestra.run.smithi086.stderr:2024-08-06T18:38:22.932+0000 7f3443fff640 0 --2- 172.21.15.86:0/2321313391 >> [v2:172.21.15.86:3300/0,v1:172.21.15.86:6789/0] conn(0x7f34441041a0 0x7f3444198a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request get_initial_auth_request returned -2 2024-08-06T18:38:22.950 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:22.951 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:22.951 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:22.968 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:22.968 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:22.968 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:22.969 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:22.969 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:22.986 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:22.986 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:22.986 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:22.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:22.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:22.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:22.987 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:23.011 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:23.145 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:23.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:23 smithi086 ceph-mon[29229]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:23.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:38:23.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:23 smithi086 ceph-mon[29229]: osdmap e852: 12 total, 3 up, 11 in 2024-08-06T18:38:23.197 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:23 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2321313391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:23.230 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:23.230 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_blocked_ops 2024-08-06T18:38:23.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:23 smithi117 ceph-mon[32835]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:38:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:23 smithi117 ceph-mon[32835]: osdmap e852: 12 total, 3 up, 11 in 2024-08-06T18:38:23.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:23 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2321313391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:23.409 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:23.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:23 smithi107 ceph-mon[32381]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:23.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1748512688' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-08-06T18:38:23.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:23 smithi107 ceph-mon[32381]: osdmap e852: 12 total, 3 up, 11 in 2024-08-06T18:38:23.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:23 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2321313391' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:23.478 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:23.540 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:23.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:23.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:23.670 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:23.806 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:23.911 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:23.912 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:23 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:23.912 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:23.913 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:23.937 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:23.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:23.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:23 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:24.068 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:24.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:24 smithi117 ceph-mon[32835]: pgmap v2499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:24.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.154 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:24 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.198 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:24.221 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:24 smithi086 ceph-mon[29229]: pgmap v2499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:24.221 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.221 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.221 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:24 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.330 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:24.461 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:24 smithi107 ceph-mon[32381]: pgmap v2499: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:24 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:24.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:24.484 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:24 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:24.593 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:24.723 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:24.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:24.899 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:24.991 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:25.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:25.199 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset nodeep-scrub 2024-08-06T18:38:25.252 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:25.388 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:25.476 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:25.485 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:25.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.11 dump_historic_ops 2024-08-06T18:38:25.521 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:25.655 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:25.730 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.c/config 2024-08-06T18:38:25.786 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:25.913 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:38:25.923 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:26.053 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:26.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:26.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:25 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:26.185 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:26.276 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:26.316 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:26.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:26 smithi117 ceph-mon[32835]: pgmap v2500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:26.447 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:26.448 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:26 smithi086 ceph-mon[29229]: pgmap v2500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:26.448 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:26.448 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:26.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:26 smithi107 ceph-mon[32381]: pgmap v2500: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:26.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:26.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:26.784 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:26.784 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:26 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:26.851 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:26.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:26.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:26 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:26.981 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:27.081 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:27.081 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:27 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:27.111 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:27.170 INFO:teuthology.orchestra.run.smithi117.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:27.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:27.373 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:27.509 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:27.641 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:27.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:27.856 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:27.856 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_ops_in_flight 2024-08-06T18:38:27.907 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:28.040 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:28.066 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:27 smithi107 ceph-mon[32381]: pgmap v2501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:28.066 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:27 smithi107 ceph-mon[32381]: 2.1 scrub starts 2024-08-06T18:38:28.066 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:27 smithi107 ceph-mon[32381]: 2.1 scrub ok 2024-08-06T18:38:28.066 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:27 smithi107 ceph-mon[32381]: osdmap e853: 12 total, 3 up, 11 in 2024-08-06T18:38:28.175 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:28.310 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:28.332 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:27 smithi117 ceph-mon[32835]: pgmap v2501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:28.332 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:27 smithi117 ceph-mon[32835]: 2.1 scrub starts 2024-08-06T18:38:28.332 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:27 smithi117 ceph-mon[32835]: 2.1 scrub ok 2024-08-06T18:38:28.332 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:27 smithi117 ceph-mon[32835]: osdmap e853: 12 total, 3 up, 11 in 2024-08-06T18:38:28.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:27 smithi086 ceph-mon[29229]: pgmap v2501: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:28.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:27 smithi086 ceph-mon[29229]: 2.1 scrub starts 2024-08-06T18:38:28.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:27 smithi086 ceph-mon[29229]: 2.1 scrub ok 2024-08-06T18:38:28.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:27 smithi086 ceph-mon[29229]: osdmap e853: 12 total, 3 up, 11 in 2024-08-06T18:38:28.441 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:28.555 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:28.576 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:28.711 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:28.792 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:28.792 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:28 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:28.843 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:28.954 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is unset 2024-08-06T18:38:28.975 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:29.040 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:28 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:29.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:29.040 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:29 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:29.041 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:29.041 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:28 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:29.106 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:29.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:28 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:29.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:28 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd={"prefix": "osd unset", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:29.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:29.373 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:29.510 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:29.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:29.532 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:29 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:29.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:29.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:29.864 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:29.864 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:29.865 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:29.883 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:29.883 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:29.884 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:29.899 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:29.899 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:29.900 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:29.902 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:30.032 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:30.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:30.184 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:30.184 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:30.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:29 smithi086 ceph-mon[29229]: pgmap v2503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:30.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:29 smithi086 ceph-mon[29229]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:30.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:30.184 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:29 smithi086 ceph-mon[29229]: osdmap e854: 12 total, 3 up, 11 in 2024-08-06T18:38:30.185 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:29 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/2575090599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:29 smithi107 ceph-mon[32381]: pgmap v2503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:29 smithi107 ceph-mon[32381]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:30.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:29 smithi107 ceph-mon[32381]: osdmap e854: 12 total, 3 up, 11 in 2024-08-06T18:38:30.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:29 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/2575090599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:30.294 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:29 smithi117 ceph-mon[32835]: pgmap v2503: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:29 smithi117 ceph-mon[32835]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-08-06T18:38:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2473662989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:29 smithi117 ceph-mon[32835]: osdmap e854: 12 total, 3 up, 11 in 2024-08-06T18:38:30.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:29 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/2575090599' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:30.425 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:30.561 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:30.581 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:30.582 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:30 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:30.690 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:30.821 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:30.950 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:31.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:30 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:30 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.217 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:31.347 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:30 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:31.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:31.619 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:31.659 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:31.660 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set noscrub 2024-08-06T18:38:31.755 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:31.886 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:31.926 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:31.967 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:31 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:32.023 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:32.158 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:32.172 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:32.173 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:31 smithi086 ceph-mon[29229]: pgmap v2505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:32.183 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:31 smithi117 ceph-mon[32835]: pgmap v2505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:32.292 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:32.318 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:31 smithi107 ceph-mon[32381]: pgmap v2505: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:32.427 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:32.428 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:32.559 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:32.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:32.823 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:32.886 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_blocked_ops 2024-08-06T18:38:32.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:33.090 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:33.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:32 smithi086 ceph-mon[29229]: osdmap e855: 12 total, 3 up, 11 in 2024-08-06T18:38:33.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:33.353 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:33.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:32 smithi117 ceph-mon[32835]: osdmap e855: 12 total, 3 up, 11 in 2024-08-06T18:38:33.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:32 smithi107 ceph-mon[32381]: osdmap e855: 12 total, 3 up, 11 in 2024-08-06T18:38:33.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:33.608 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:33.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:33.661 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-08-06T18:38:33.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:33.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:34.021 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:34.045 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:33 smithi117 ceph-mon[32835]: pgmap v2507: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:34.154 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:34.257 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:33 smithi086 ceph-mon[29229]: pgmap v2507: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:34.290 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:34.421 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:34.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:33 smithi107 ceph-mon[32381]: pgmap v2507: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:34.555 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:34.686 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:34.817 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:34.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:34.839 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:34.948 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:35.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:35.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:35.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:34 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:35.219 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:35.350 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:35.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:35.481 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:35.591 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:35.591 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:35 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:35.618 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:35.752 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:35.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:36.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:36.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:36.163 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:35 smithi086 ceph-mon[29229]: pgmap v2508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:36.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:35 smithi107 ceph-mon[32381]: pgmap v2508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:36.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:36.213 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:35 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:36.262 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:36.292 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:36.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:35 smithi117 ceph-mon[32835]: pgmap v2508: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:36.423 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:36.484 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:36.558 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:36.607 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:36.608 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:36.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:36.830 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:36.961 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:36.991 INFO:teuthology.orchestra.run.smithi086.stderr:noscrub is set 2024-08-06T18:38:37.097 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:37.103 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.3 dump_historic_ops 2024-08-06T18:38:37.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:36 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:37.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:37.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:36 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:37.234 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:37.259 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:36 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:37.369 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:37.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:36 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd={"prefix": "osd set", "key": "noscrub"} : dispatch 2024-08-06T18:38:37.504 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:37.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:37.661 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:37.661 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:37 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:37.771 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:37.793 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:37.793 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:37.903 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:37.974 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:38.034 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:38.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:37 smithi086 ceph-mon[29229]: pgmap v2509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:38.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:37 smithi086 ceph-mon[29229]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:38.056 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:37 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:38.057 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:37 smithi086 ceph-mon[29229]: osdmap e856: 12 total, 3 up, 11 in 2024-08-06T18:38:38.057 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:38.057 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:38.057 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:38.057 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:37 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:38.164 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:38.300 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:38.326 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:37 smithi107 ceph-mon[32381]: pgmap v2509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:38.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:37 smithi107 ceph-mon[32381]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:38.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:37 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:38.327 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:37 smithi107 ceph-mon[32381]: osdmap e856: 12 total, 3 up, 11 in 2024-08-06T18:38:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:37 smithi117 ceph-mon[32835]: pgmap v2509: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:37 smithi117 ceph-mon[32835]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:37 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/3708480940' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-08-06T18:38:38.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:37 smithi117 ceph-mon[32835]: osdmap e856: 12 total, 3 up, 11 in 2024-08-06T18:38:38.436 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:38.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:38.706 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:38.842 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:38.973 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:39.108 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:39.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:39.115 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:38 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:39.243 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:39.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:39.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:39 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:39.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:39.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:39.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:39.644 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:39.777 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:39.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:39.990 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:39.990 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:39 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:40.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:40.176 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:40.246 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:39 smithi086 ceph-mon[29229]: pgmap v2511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:40.247 INFO:teuthology.orchestra.run.smithi086.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-08-06T18:38:40.312 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:40.338 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:39 smithi117 ceph-mon[32835]: pgmap v2511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:40.447 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:40.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd set nodeep-scrub 2024-08-06T18:38:40.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:39 smithi107 ceph-mon[32381]: pgmap v2511: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:40.582 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:40.716 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:40.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:40.983 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:40.997 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:40.998 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:40 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-08-06T18:38:40.999 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_ops_in_flight 2024-08-06T18:38:41.118 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:41.253 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:41.387 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:41.430 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:41.520 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:41.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:41.700 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:41.700 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:41.700 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:41.700 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:41 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:41.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:41.924 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:41.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:41.962 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:41 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:42.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:42.193 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.1 2024-08-06T18:38:42.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:41 smithi086 ceph-mon[29229]: pgmap v2512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:42.252 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:41 smithi086 ceph-mon[29229]: osdmap e857: 12 total, 3 up, 11 in 2024-08-06T18:38:42.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:42.353 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:41 smithi117 ceph-mon[32835]: pgmap v2512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:42.354 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:41 smithi117 ceph-mon[32835]: osdmap e857: 12 total, 3 up, 11 in 2024-08-06T18:38:42.462 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:41 smithi107 ceph-mon[32381]: pgmap v2512: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:42.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:41 smithi107 ceph-mon[32381]: osdmap e857: 12 total, 3 up, 11 in 2024-08-06T18:38:42.577 INFO:teuthology.orchestra.run.smithi086.stdout:osd.0: {} 2024-08-06T18:38:42.578 INFO:teuthology.orchestra.run.smithi086.stderr:osd.0: osd_enable_op_tracker = '' 2024-08-06T18:38:42.578 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.1 2024-08-06T18:38:42.582 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:42.594 INFO:teuthology.orchestra.run.smithi086.stdout:osd.2: {} 2024-08-06T18:38:42.594 INFO:teuthology.orchestra.run.smithi086.stderr:osd.2: osd_enable_op_tracker = '' 2024-08-06T18:38:42.594 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.3 2024-08-06T18:38:42.594 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.4 2024-08-06T18:38:42.594 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.5 2024-08-06T18:38:42.598 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:42.612 INFO:teuthology.orchestra.run.smithi086.stdout:osd.6: {} 2024-08-06T18:38:42.613 INFO:teuthology.orchestra.run.smithi086.stderr:osd.6: osd_enable_op_tracker = '' 2024-08-06T18:38:42.613 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-08-06T18:38:42.613 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.8 2024-08-06T18:38:42.613 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.9 2024-08-06T18:38:42.614 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.10 2024-08-06T18:38:42.614 INFO:teuthology.orchestra.run.smithi086.stderr:Error ENXIO: problem getting command descriptions from osd.11 2024-08-06T18:38:42.732 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:42.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:43.001 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:43 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/269253040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:43.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:43 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:43.132 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:43.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:43.156 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:42 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:43.156 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:43 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/269253040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:43.157 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:43 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:43.265 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:43.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:43 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/269253040' entity='client.admin' cmd={"prefix": "osd ls"} : dispatch 2024-08-06T18:38:43.374 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:43 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:43.397 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:43.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:43 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:43.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:43.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:43.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:43.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:43.615 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:43 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:43.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:43.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:43.932 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:44.068 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:44.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:44 smithi086 ceph-mon[29229]: pgmap v2514: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:44.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.105 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:44 smithi086 ceph-mon[29229]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:44.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:44.212 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:44 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:44 smithi107 ceph-mon[32381]: pgmap v2514: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:44.212 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.213 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:44 smithi107 ceph-mon[32381]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.336 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:44.357 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:44.358 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:44 smithi117 ceph-mon[32835]: pgmap v2514: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:44 smithi117 ceph-mon[32835]: osd_map_cache_size (1) is not > osd_pg_epoch_persisted_max_stale (40) 2024-08-06T18:38:44.466 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:44.602 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:44.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:44.615 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:44.626 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-08-06T18:38:44.627 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-08-06T18:38:44.737 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:44.869 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:44.884 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:44.884 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:44 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:45.005 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:45.140 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:45.275 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:45.411 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:45.547 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:45.683 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:45.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:45.940 INFO:teuthology.orchestra.run.smithi086.stdout:ERROR: (22) Invalid argument 2024-08-06T18:38:45.940 INFO:teuthology.orchestra.run.smithi086.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-08-06T18:38:45.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:46.003 INFO:teuthology.orchestra.run.smithi086.stderr:nodeep-scrub is set 2024-08-06T18:38:46.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:46.220 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:46.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:46.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:46 smithi086 ceph-mon[29229]: pgmap v2515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:46.365 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:46 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:46 smithi117 ceph-mon[32835]: pgmap v2515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:46.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:46 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:46.462 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:46 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:46 smithi107 ceph-mon[32381]: pgmap v2515: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:46.463 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:46 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd={"prefix": "osd set", "key": "nodeep-scrub"} : dispatch 2024-08-06T18:38:46.488 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.11 2024-08-06T18:38:46.624 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:46.628 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-08-06T18:38:46.650 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:46.650 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:46.759 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:46.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:46.916 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:46.916 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:46 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:47.024 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:47.160 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:47.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:47 smithi086 ceph-mon[29229]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:47.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:47 smithi086 ceph-mon[29229]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:47.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:47 smithi086 ceph-mon[29229]: osdmap e858: 12 total, 3 up, 11 in 2024-08-06T18:38:47.182 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:47 smithi086 ceph-mon[29229]: osdmap e859: 12 total, 3 up, 11 in 2024-08-06T18:38:47.290 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:47.316 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:47.317 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:47.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:47 smithi107 ceph-mon[32381]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:47.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:47 smithi107 ceph-mon[32381]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:47.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:47 smithi107 ceph-mon[32381]: osdmap e858: 12 total, 3 up, 11 in 2024-08-06T18:38:47.317 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:47 smithi107 ceph-mon[32381]: osdmap e859: 12 total, 3 up, 11 in 2024-08-06T18:38:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:47 smithi117 ceph-mon[32835]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-08-06T18:38:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:47 smithi117 ceph-mon[32835]: from='client.? 172.21.15.86:0/1553252170' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-08-06T18:38:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:47 smithi117 ceph-mon[32835]: osdmap e858: 12 total, 3 up, 11 in 2024-08-06T18:38:47.373 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:47 smithi117 ceph-mon[32835]: osdmap e859: 12 total, 3 up, 11 in 2024-08-06T18:38:47.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6 2024-08-06T18:38:47.429 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_blocked_ops 2024-08-06T18:38:47.450 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:47.450 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:47 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:47.514 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:47.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:47.696 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48450 (podman-init) on client request. 2024-08-06T18:38:47.712 INFO:journalctl@ceph.osd.6.smithi107.stdout:Aug 06 18:38:47 smithi107 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.6.service: Sending signal SIGHUP to process 48452 (ceph-osd) on client request. 2024-08-06T18:38:47.831 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.8 2024-08-06T18:38:47.967 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:48.101 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.5 2024-08-06T18:38:48.115 INFO:journalctl@ceph.mon.a.smithi086.stdout:Aug 06 18:38:48 smithi086 ceph-mon[29229]: pgmap v2517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:48.123 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:48 smithi117 ceph-mon[32835]: pgmap v2517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:48.187 INFO:teuthology.orchestra.run.smithi086.stderr:Inferring config /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/mon.a/config 2024-08-06T18:38:48.237 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.9 2024-08-06T18:38:48.372 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:48.462 INFO:journalctl@ceph.mon.b.smithi107.stdout:Aug 06 18:38:48 smithi107 ceph-mon[32381]: pgmap v2517: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:48.509 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:48.591 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43627 (podman-init) on client request. 2024-08-06T18:38:48.592 INFO:journalctl@ceph.osd.0.smithi086.stdout:Aug 06 18:38:48 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0.service: Sending signal SIGHUP to process 43629 (ceph-osd) on client request. 2024-08-06T18:38:48.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.4 2024-08-06T18:38:48.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.7 2024-08-06T18:38:48.822 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub 2024-08-06T18:38:48.914 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.10 2024-08-06T18:38:49.050 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2 2024-08-06T18:38:49.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55037 (podman-init) on client request. 2024-08-06T18:38:49.115 INFO:journalctl@ceph.osd.2.smithi086.stdout:Aug 06 18:38:49 smithi086 systemd[1]: ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.2.service: Sending signal SIGHUP to process 55039 (ceph-osd) on client request. 2024-08-06T18:38:49.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.3 2024-08-06T18:38:49.311 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0 2024-08-06T18:38:49.350 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T18:38:49.351 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1370, in do_optrack_toggle self.ceph_manager.inject_args('osd', '*', File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2068, in inject_args self.raw_cluster_cmd('--', 'tell', whom, 'injectargs', opt_arg) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true" 2024-08-06T18:38:49.352 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1370, in do_optrack_toggle self.ceph_manager.inject_args('osd', '*', File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2068, in inject_args self.raw_cluster_cmd('--', 'tell', whom, 'injectargs', opt_arg) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: "sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true" 2024-08-06T18:38:49.352 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T18:38:49.353 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph daemon osd.0 dump_historic_ops 2024-08-06T18:38:49.353 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T18:38:49.354 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1412, in do_noscrub_toggle self.ceph_manager.raw_cluster_cmd('osd', 'unset', 'noscrub') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub' 2024-08-06T18:38:49.354 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1412, in do_noscrub_toggle self.ceph_manager.raw_cluster_cmd('osd', 'unset', 'noscrub') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 179, in _raise_for_status raise CommandCrashedError(command=self.command) teuthology.exceptions.CommandCrashedError: Command crashed: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:93ef537d2bb6d0c4eb0443bb47a842ef98f1e3a1 shell --fsid fd7e697e-541a-11ef-bccc-c7b262605968 -- ceph osd unset noscrub' 2024-08-06T18:38:49.354 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T18:38:49.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: SSH connection to smithi086 was lost: 'sudo systemctl kill -s 1 ceph-fd7e697e-541a-11ef-bccc-c7b262605968@osd.0' 2024-08-06T18:38:49.357 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1390, in do_dump_ops self.ceph_manager.osd_admin_socket(osd, command=['dump_historic_ops'], File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1870, in osd_admin_socket return self.admin_socket('osd', osd_id, command, check_status, timeout, stdout) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1898, in admin_socket return shell( File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 310, in read_all raise EOFError() EOFError 2024-08-06T18:38:49.357 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 190, in wrapper return func(self) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1390, in do_dump_ops self.ceph_manager.osd_admin_socket(osd, command=['dump_historic_ops'], File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1870, in osd_admin_socket return self.admin_socket('osd', osd_id, command, check_status, timeout, stdout) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1898, in admin_socket return shell( File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/virtualenv/lib/python3.10/site-packages/paramiko/packet.py", line 310, in read_all raise EOFError() EOFError 2024-08-06T18:38:49.456 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:38:49.457 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:38:49.483 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:38:50.275 INFO:journalctl@ceph.mon.c.smithi117.stdout:Aug 06 18:38:50 smithi117 ceph-mon[32835]: pgmap v2519: 33 pgs: 2 active+clean, 5 stale+undersized+peered, 1 stale+active+clean, 13 undersized+peered, 12 unknown; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-08-06T18:38:51.329 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-08-06T18:38:51.330 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi117 was lost: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" 2024-08-06T18:38:51.566 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3238e92495dc447ba34b41b3b1431533 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/exec.py", line 66, in task remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 174, in _raise_for_status raise ConnectionLostError(command=self.command, teuthology.exceptions.ConnectionLostError: SSH connection to smithi117 was lost: "sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror'" 2024-08-06T18:38:51.568 DEBUG:teuthology.run_tasks:Unwinding manager thrashosds 2024-08-06T18:38:51.576 INFO:tasks.thrashosds:joining thrashosds 2024-08-06T18:38:51.576 INFO:tasks.thrashosds.thrasher:joining the do_sighup greenlet 2024-08-06T18:38:58.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:38:58.491 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:39:58.519 DEBUG:teuthology.orchestra.remote:timed out 2024-08-06T18:40:10.529 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:40:10.531 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:40:13.630 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:40:16.633 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:40:16.634 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:40:16.702 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:40:16.703 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-08-06T18:40:16.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Failed to reconnect to smithi086 2024-08-06T18:40:16.804 INFO:tasks.thrashosds.thrasher:joining the do_optrack_toggle greenlet 2024-08-06T18:40:16.804 INFO:tasks.thrashosds.thrasher:joining the do_dump_ops greenlet 2024-08-06T18:40:16.804 INFO:tasks.thrashosds.thrasher:joining the do_noscrub_toggle greenlet 2024-08-06T18:40:16.804 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-08-06T18:40:16.805 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:40:16.805 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:40:35.295 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:40:44.302 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:40:44.304 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:40:46.176 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:40:58.189 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:40:58.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:40:58.334 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:41:02.485 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:41:09.631 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:41:19.794 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:41:32.940 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:41:49.084 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:42:08.227 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:42:30.374 ERROR:teuthology.orchestra.connection:Error authenticating with smithi086.front.sepia.ceph.com: AuthenticationException('Authentication failed.') 2024-08-06T18:43:55.388 DEBUG:teuthology.orchestra.remote:timed out 2024-08-06T18:43:58.390 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:43:58.391 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:44:16.735 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:44:16.735 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-08-06T18:44:16.736 ERROR:teuthology.run_tasks:Manager failed: thrashosds Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-08-06T18:44:16.750 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-08-06T18:44:16.758 INFO:tasks.cephadm:Teardown begin 2024-08-06T18:44:16.759 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 2319, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-08-06T18:44:16.764 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:44:16.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:44:19.838 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-08-06T18:44:28.839 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:44:28.840 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:44:29.090 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-08-06T18:44:29.405 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-08-06T18:44:29.405 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-06T18:44:29.552 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T18:44:29.553 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:44:45.305 DEBUG:teuthology.orchestra.run.smithi107:> true 2024-08-06T18:44:45.673 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi107.front.sepia.ceph.com' 2024-08-06T18:44:45.673 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-06T18:44:45.746 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T18:44:45.746 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-08-06T18:44:45.825 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-08-06T18:44:45.908 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-08-06T18:44:45.908 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-08-06T18:44:45.971 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-08-06T18:44:45.971 DEBUG:teuthology.orchestra.run.smithi086:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-06T18:44:45.989 DEBUG:teuthology.orchestra.run.smithi107:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-06T18:44:46.004 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-08-06T18:44:46.025 INFO:tasks.cephadm:Stopping all daemons... 2024-08-06T18:44:46.025 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-08-06T18:44:46.025 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a 2024-08-06T18:44:46.062 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service: Unit ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a.service not loaded. 2024-08-06T18:44:46.063 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-08-06T18:44:46.064 ERROR:tasks.cephadm:Failed to stop "ceph.mon.a" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 845, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 2319, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 864, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/daemon/cephadmunit.py", line 167, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 5: 'sudo systemctl stop ceph-fd7e697e-541a-11ef-bccc-c7b262605968@mon.a' 2024-08-06T18:44:46.078 INFO:tasks.cephadm:Archiving crash dumps... 2024-08-06T18:44:46.080 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi086/crash 2024-08-06T18:44:46.081 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash -- . 2024-08-06T18:44:46.158 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash: Cannot open: No such file or directory 2024-08-06T18:44:46.159 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.160 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi107/crash 2024-08-06T18:44:46.161 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash -- . 2024-08-06T18:44:46.203 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash: Cannot open: No such file or directory 2024-08-06T18:44:46.203 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.205 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi117/crash 2024-08-06T18:44:46.206 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash -- . 2024-08-06T18:44:46.235 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/crash: Cannot open: No such file or directory 2024-08-06T18:44:46.236 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.237 INFO:tasks.cephadm:Checking cluster log for badness... 2024-08-06T18:44:46.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(REQUEST_STUCK\)' | egrep -v '\(MON_DOWN\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v osd_map_cache_size | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | head -n 1 2024-08-06T18:44:46.267 INFO:teuthology.orchestra.run.smithi086.stderr:grep: /var/log/ceph/fd7e697e-541a-11ef-bccc-c7b262605968/ceph.log: No such file or directory 2024-08-06T18:44:46.275 INFO:tasks.cephadm:Compressing logs... 2024-08-06T18:44:46.275 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-06T18:44:46.318 DEBUG:teuthology.orchestra.run.smithi107:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-06T18:44:46.321 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-08-06T18:44:46.339 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/ceph’: No such file or directory 2024-08-06T18:44:46.339 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-06T18:44:46.341 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-08-06T18:44:46.341 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m0.011s 2024-08-06T18:44:46.341 INFO:teuthology.orchestra.run.smithi086.stderr:user 0m0.006s 2024-08-06T18:44:46.342 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m0.013s 2024-08-06T18:44:46.343 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/ceph’: No such file or directory 2024-08-06T18:44:46.343 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-06T18:44:46.345 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-08-06T18:44:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:real 0m0.011s 2024-08-06T18:44:46.345 INFO:teuthology.orchestra.run.smithi117.stderr:user 0m0.010s 2024-08-06T18:44:46.346 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m0.009s 2024-08-06T18:44:46.370 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/ceph’: No such file or directory 2024-08-06T18:44:46.371 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-08-06T18:44:46.373 INFO:teuthology.orchestra.run.smithi107.stderr: 2024-08-06T18:44:46.373 INFO:teuthology.orchestra.run.smithi107.stderr:real 0m0.043s 2024-08-06T18:44:46.373 INFO:teuthology.orchestra.run.smithi107.stderr:user 0m0.010s 2024-08-06T18:44:46.373 INFO:teuthology.orchestra.run.smithi107.stderr:sys 0m0.012s 2024-08-06T18:44:46.373 INFO:tasks.cephadm:Archiving logs... 2024-08-06T18:44:46.374 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi086/log 2024-08-06T18:44:46.374 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-06T18:44:46.404 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2024-08-06T18:44:46.404 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.405 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi107/log 2024-08-06T18:44:46.406 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-06T18:44:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2024-08-06T18:44:46.429 INFO:teuthology.orchestra.run.smithi107.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.430 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi117/log 2024-08-06T18:44:46.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-08-06T18:44:46.456 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/log/ceph: Cannot open: No such file or directory 2024-08-06T18:44:46.456 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:46.457 INFO:tasks.cephadm:Removing cluster... 2024-08-06T18:44:46.457 DEBUG:teuthology.orchestra.run.smithi086:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fd7e697e-541a-11ef-bccc-c7b262605968 --force 2024-08-06T18:44:46.477 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: /home/ubuntu/cephtest/cephadm: command not found 2024-08-06T18:44:46.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:46.478 INFO:tasks.cephadm:Teardown complete 2024-08-06T18:44:46.478 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 845, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 2319, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 2280, in task with contextutil.nested( File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 1909, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 229, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/cephadm.py", line 383, in _rm_cluster remote.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid fd7e697e-541a-11ef-bccc-c7b262605968 --force' 2024-08-06T18:44:46.496 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-08-06T18:44:46.505 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/install/__init__.py", line 641, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 2024-08-06T18:44:46.519 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-08-06T18:44:46.519 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-06T18:44:46.522 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-06T18:44:46.527 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-08-06T18:44:46.648 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-08-06T18:44:46.656 INFO:teuthology.task.clock:Checking final clock skew... 2024-08-06T18:44:46.657 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T18:44:46.659 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T18:44:46.661 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-08-06T18:44:46.671 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-08-06T18:44:46.673 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: ntpq: command not found 2024-08-06T18:44:46.674 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-08-06T18:44:46.675 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T18:44:46.675 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-08-06T18:44:46.675 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 3 6 17 16 +3305us[+3383us] +/- 65ms 2024-08-06T18:44:46.676 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 17 16 -774us[ -696us] +/- 56ms 2024-08-06T18:44:46.676 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 17 16 -2708us[-2708us] +/- 69ms 2024-08-06T18:44:46.676 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T18:44:46.677 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T18:44:46.677 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2024-08-06T18:44:46.678 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv01.front.sepia.ceph.com 3 6 17 3 +3314us[+3336us] +/- 65ms 2024-08-06T18:44:46.678 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 3 6 17 3 -732us[ -711us] +/- 56ms 2024-08-06T18:44:46.678 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 17 3 -2660us[-2660us] +/- 69ms 2024-08-06T18:44:46.678 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T18:44:46.679 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-08-06T18:44:46.679 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-08-06T18:44:46.680 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv01.front.sepia.ceph.com 3 6 17 56 +3352us[+3374us] +/- 64ms 2024-08-06T18:44:46.680 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 17 56 -709us[ -687us] +/- 55ms 2024-08-06T18:44:46.680 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 17 55 -2680us[-2680us] +/- 68ms 2024-08-06T18:44:46.680 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 7 0 - +0ns[ +0ns] +/- 0ns 2024-08-06T18:44:46.680 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-08-06T18:44:46.689 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-08-06T18:44:46.689 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-08-06T18:44:46.697 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-08-06T18:44:46.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:46.730 INFO:teuthology.orchestra.run.smithi086.stderr:mkdir: cannot create directory ‘/home/ubuntu/cephtest/archive/audit’: No such file or directory 2024-08-06T18:44:46.730 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/selinux.py", line 166, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/selinux.py", line 196, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log' 2024-08-06T18:44:46.738 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-08-06T18:44:46.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-08-06T18:44:46.775 INFO:teuthology.task.internal:Duration was 4228.059889 seconds 2024-08-06T18:44:46.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-08-06T18:44:46.783 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-08-06T18:44:46.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-06T18:44:46.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-06T18:44:46.790 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-08-06T18:44:46.826 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T18:44:46.831 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T18:44:46.834 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-08-06T18:44:47.118 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-08-06T18:44:47.118 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-08-06T18:44:47.119 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-06T18:44:47.137 INFO:teuthology.orchestra.run.smithi086.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2024-08-06T18:44:47.139 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2024-08-06T18:44:47.140 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-06T18:44:47.161 INFO:teuthology.orchestra.run.smithi107.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2024-08-06T18:44:47.162 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-08-06T18:44:47.162 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-08-06T18:44:47.191 INFO:teuthology.orchestra.run.smithi117.stderr:grep: /home/ubuntu/cephtest/archive/syslog/kern.log: No such file or directory 2024-08-06T18:44:47.192 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-08-06T18:44:47.192 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-06T18:44:47.195 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-06T18:44:47.202 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-08-06T18:44:47.207 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2024-08-06T18:44:47.215 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2024-08-06T18:44:47.246 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/home/ubuntu/cephtest/archive/syslog’: No such file or directory 2024-08-06T18:44:47.255 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-08-06T18:44:47.256 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-06T18:44:47.259 DEBUG:teuthology.orchestra.run.smithi107:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-06T18:44:47.268 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-08-06T18:44:47.271 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2024-08-06T18:44:47.280 INFO:teuthology.orchestra.run.smithi107.stderr:bash: line 1: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2024-08-06T18:44:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:47.297 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/syslog.py", line 185, in syslog run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz' 2024-08-06T18:44:47.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-08-06T18:44:47.319 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-08-06T18:44:47.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-08-06T18:44:47.323 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz: No such file or directory 2024-08-06T18:44:47.345 INFO:teuthology.orchestra.run.smithi086.stderr:mv: cannot stat '/etc/sudoers.orig.teuthology': No such file or directory 2024-08-06T18:44:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:47.347 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers' 2024-08-06T18:44:47.360 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-08-06T18:44:47.369 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-06T18:44:47.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-06T18:44:47.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-08-06T18:44:47.417 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-08-06T18:44:47.422 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-08-06T18:44:47.424 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2024-08-06T18:44:47.427 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2024-08-06T18:44:47.430 INFO:teuthology.orchestra.run.smithi117.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2024-08-06T18:44:47.432 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2024-08-06T18:44:47.435 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest/archive/coredump’: No such file or directory 2024-08-06T18:44:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:47.437 INFO:teuthology.orchestra.run.smithi086.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2024-08-06T18:44:47.437 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 491, in coredump run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: "sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump" 2024-08-06T18:44:47.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-08-06T18:44:47.461 INFO:teuthology.task.internal:Transferring archived files... 2024-08-06T18:44:47.462 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/lflores-2024-08-06_17:19:44-rados:thrash-old-clients-main-distro-default-smithi/7840173/remote/smithi086 2024-08-06T18:44:47.462 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-08-06T18:44:47.467 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/archive/coredump': No such file or directory 2024-08-06T18:44:47.499 INFO:teuthology.orchestra.run.smithi086.stderr:tar: /home/ubuntu/cephtest/archive: Cannot open: No such file or directory 2024-08-06T18:44:47.499 INFO:teuthology.orchestra.run.smithi086.stderr:tar: Error is not recoverable: exiting now 2024-08-06T18:44:47.500 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/misc.py", line 736, in pull_directory tar = tarfile.open(mode='r|', fileobj=r.stdout) File "/usr/lib/python3.10/tarfile.py", line 1836, in open t = cls(name, filemode, stream, **kwargs) File "/usr/lib/python3.10/tarfile.py", line 1714, in __init__ self.firstmember = self.next() File "/usr/lib/python3.10/tarfile.py", line 2617, in next raise ReadError("empty file") from None tarfile.ReadError: empty file 2024-08-06T18:44:47.508 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-08-06T18:44:47.518 INFO:teuthology.task.internal:Not uploading archives. 2024-08-06T18:44:47.518 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-08-06T18:44:47.527 INFO:teuthology.task.internal:Tidying up after the test... 2024-08-06T18:44:47.527 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-06T18:44:47.543 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-06T18:44:47.545 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-08-06T18:44:47.554 INFO:teuthology.orchestra.run.smithi086.stderr:find: ‘/home/ubuntu/cephtest’: No such file or directory 2024-08-06T18:44:47.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-08-06T18:44:47.555 INFO:teuthology.orchestra.run.smithi086.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': No such file or directory 2024-08-06T18:44:47.556 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/thrashosds.py", line 218, in task cluster_manager.wait_for_all_osds_up() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2861, in wait_for_all_osds_up while not self.are_all_osds_up(): File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2851, in are_all_osds_up x = self.get_osd_dump() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2614, in get_osd_dump return self.get_osd_dump_json()['osds'] File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 2606, in get_osd_dump_json out = self.raw_cluster_cmd('osd', 'dump', '--format=json') File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1661, in raw_cluster_cmd return self.run_cluster_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 1639, in run_cluster_cmd return shell(self.ctx, self.cluster, self.controller, File "/home/teuthworker/src/github.com_ceph_ceph-c_0325fd3c11a225c0fd9eb38fab88aad1370ad779/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi086 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__ self.gen.throw(typ, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_4bc527c722abc094dbc33e04519a3647224076f6/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi086 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2024-08-06T18:44:47.568 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-08-06T18:44:47.577 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-08-06T18:44:47.588 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: ConnectionError: Failed to reconnect to smithi086 2024-08-06T18:44:47.591 INFO:teuthology.run:Summary data: description: rados:thrash-old-clients/{0-distro$/{centos_9.stream} 0-size-min-size-overrides/2-size-2-min-size 1-install/quincy backoff/normal ceph clusters/{openstack three-plus-one} d-balancer/crush-compat mon_election/classic msgr-failures/fastclose rados thrashers/mapgap thrashosds-health workloads/rbd_cls} duration: 4228.059889316559 failure_reason: 'SSH connection to smithi117 was lost: "sudo TESTDIR=/home/ubuntu/cephtest bash -c ''ceph_test_cls_rbd --gtest_filter=-TestClsRbd.get_features:TestClsRbd.parents:TestClsRbd.mirror''"' flavor: default owner: scheduled_lflores@folio02 sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3238e92495dc447ba34b41b3b1431533 status: fail success: false 2024-08-06T18:44:47.592 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-08-06T18:44:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:find: ‘/home/ubuntu/cephtest’: No such file or directory 2024-08-06T18:44:47.608 INFO:teuthology.orchestra.run.smithi107.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': No such file or directory 2024-08-06T18:44:47.609 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/home/ubuntu/cephtest’: No such file or directory 2024-08-06T18:44:47.609 INFO:teuthology.orchestra.run.smithi117.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': No such file or directory 2024-08-06T18:44:47.685 INFO:teuthology.run:FAIL